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A LEVEL OF CARE INDEX: DEVELOPMENT AND COMPARATIVE 
"EVALUATION OF A COMPUTER-AIDED UTILIZATION REVIEW SELECTOR. METHOD ee 


Rebecca Fuhrer, M.Sc. 
The purpose of this study has been to develop and evaluate a 


computerized decision model to select potentially inappropriately » 


located acute care aedeagus patients for the utilization review 


fvaction of the Professional Standards Review Organization (PSRO). 


A hospital's Utilization Review Committee can be accorded 
deldgatéd PSRO status” if chn demonstrate continued effectiveness in 


meeting its objectives of assuring | appropriate acilination while 


ens#fring the quality of care. By focusing on the procedure for patient 
selection, the. effectiveness as well as efficiency of utilization review 


should be ‘enhanced. 


Most for the above referred to. patients 


“ 


are based on. the use of length of ssid = with the expectation that 


changes in level of care need will stingy with certain time 


norms. - 
| An alternative selection procedure based on the medical services 


required by..the. is proposed. ‘Statistical methodology (Multiple 


| Logistic Discrimination Method) has been used to develop, a mathematical, 


capable of differentiating patients requiring acute care hospital 


ra | level of care from those who potentially do not. A matical services 
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the subjectively defjned medical’ services classification and 


: Classification system which encodes fhe health care environment in level: . 


of care terms has also been developed. It is used by the discriminant 


analysis to determine the subset of optimally discriminating variables 


for ‘acute care hospital level of care assessment. 
The decision model, is significantly better at 

predicting thé patients vino Should be selected for revi ey and the 

patients who need not be reviewed than the length of stay guidelines 


proposed by the PSRO. . The costs of each selection method need to be. 


0 \ 


fhe combined effort of the expertise of health care professionals | 


Clustering, and the power inherent in the statistical methodology used 


to empirically differentiate patterns, has yielded a mathematical model : 
which approximates a complex decision process. ae 
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_ CHAPTER I INTRODUCTION 


I.0 Introduction 


The objective of this dissertation is the development and 
evaluation-of a computerized decision model to aid in the utilization 
review the Professional Standards Review nation (PSRO) 
program (PL92-603). The sebection of patients for review hao a direct 


impact on the ‘benefits ‘and costs of the PSRO program, and has been the 


subject of many research endeavors. , A selector model is a procedure 


or algorithm which determines the sample of patients in the hospitalized 


population who should be reviewed. Most selector models, computer based 
and otherwise, —eor of ne information as an indicator of which 
patients to review. ‘The increased use of computer technology to support 
the selection gives rise to che lity of developing more 
effective selector models based on additional or other information. 
Decision modelling is the approach which: is used in this research study 

to develop a computerized selector model. 

This chapter will (1) introduce the reader to PSROs and 
utilization review, (2). discuss the aoa 9 process to arrive at 
eit hina review decisions, (3) describe a concurrent utilization 


review system, ‘and (4) @scribe- aPSRO recommended selector model and 


PROPOSE a level “of care decision model for eéisition patients for 


utidization review. 
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‘programs and make judgment -on the medical necessity and auality of ‘ae 


two basic purposes: (1). to assure the quality of care, and (2) to 
improve quality of care by identifying problems and correcting them 
through a — educational 


of care, in this poo ae is defined as the provision of the ‘appropria _ 


at the in the amounts, at the 


care, hospitals and physicians have focused almost exClusively upon ~ 


I.1 “PSROs and Utilization Review 


The Professional Standards Review Organization program was 
enacted on October 30, 1972, -as part of the 1972 Amendment to the Social 


Security Act. The PSRO Program Manual, published by the Department of ; 


Health, Education, and Welfare, on March 15, 1974, specified that the 


responsibility. PSROs would be to "review the health care 


patients under the Medicare, Medicaid, and Maternal and. Child Health 
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care rendered to Mesh patients: In addition, PSROs are to determine 
whether care is proposed to be provided or has been provided at a level 
of care which is most economical yet consistent with the patient's 


medical care needs".° me the PSRO program is intended to accomplish 


assure the appropriate utilization of health care services, PSROS und 


on the principles of peer review and the use-of explicit criteria to 


Since the PSRO Manual has made no precise definition of. quality 


appropriate level of care. Tra ‘tionally, when measuring the quality of 


medical audit. This focus, while assuring that the services delivered 
to a patient are appropriate for his/her diagnosis or medical problem, 
devotes little systematic attention to the location of these services. 


If a patient is inapprepriately located at an acute level of Care, 
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quality of care is adversely affected eis delivery, amount, and 
timing of services received by the patient are not consistent with his/ 
her medical care needs, Patient location is an in - | 
aeasubing Sala ¥> of care, in addition to its patent relevance to cost 
of care, and is of primary imporgance to utilization review. : 

The PSRO. objective of assuring the appropriate utilization of 
health care services, is an iciicatclas al previously legislated 
utilization review functions. Technically speaking, utilization review 
has been in existence since the enactment of the Medicare legislation in 
1965, but now the PSRO "retains in ae for assuring the con- 
tinued effectiveness of that review" "Utilization as | 
by the Joint Commission of Hospitals, "is 


form of peer review or medical care evaluation for the intended purpose | 


of periodic review of the utilization of the facilities and of ‘the 6 


diagnostic, nursing and therapeutic resources of the eniek ele 
pemeck to both the aivailability of their resources to all patients in 
with their care needs and the recogni tion’of medical 
8 
| Prior to the PSRO Lesisiation, reviews usually were conducted on re . 
a retrospective basis. The PSRO program, however, requires two types of 
reviews to be that is, while the patient is in 
the Concurrent revieys provide the for corrective 
action when, it is necessary, which was mae the case for retrodpentive 
reviews. The concurrent reviews consist of Admission Certification | 


assure the need for to an.acute care hospital 


‘and Continued Stay Reviews which assure that the patient continues to 
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require the services of an acute hospital. 


The legislati encourages that the responsibility for these 


evel. Hence, a hospital's utilization review 


committee can apply for delegated status, i.e., assume the respon- 
sibility for ongoing reviews, and the PSRO will delegate Said 


responsibility 1f the committee has demonstrated that it is effective 


in its review process. The PSRO periodically. assesses the continued 


‘effectiveness of delegated utilization review committees. If the 


Committee's performance is considered inadequate the delegated status - 


can be rescinded, resulting in external jurisdiction for the hospital. 


I.2 Utilization Review Objective - A Level of Care Decision 


To. satisfy the goals of utilization review, the utilization 
review committee must determine the most economical level of care 


J 


facility which Can provide the medical services a patient needs. It is 


to determine if a patient is appropriately located in an acute hospital. . 
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According to Medicare guidelines, a — is appropriately 


located in ‘hospital facility if the following statements about 


the patient are applicable: 


a. The services the patient is receiving cannot 
ordinarily be, safely and\adequately performed 
by the average, nonmedical, rational person e 
without direct supervision of trained medical 
or paramedical personnel, ; , 


and 


be The patient requires direct skilled nursing 
services (excluding observation) Prequentay, 
1e@€.e, at least once a day, 


‘the unstabilized condition of the patient. | } 
requires the skills of a nurse to detect and 3 ’ 

evaluate the patient's need for possible 5 | 
modification of treatment or institution of 

medical procecures, 


c. The patrent requires constant availability 
of medical services provided by a hospital | 
gre and not available in skilled nursing faci¥ities.? . 


“Thus, a patient is considered appropriately ‘ocated in an acute 


“hospital: ‘if ieyne is receiving’a level of care which can only be 


furnished ina hospital. The level of care is aiiiieias by the type, 


number, and/or intensity. of a combination of physician, skilled nursing, 
and ancillary services a patient is receiving. If it is determined that 
a patient requires a level of care available in a skilled nursing 


facility or other non-hospital setting, the patient can be considered to 


: : 
be misutilizing the hospital setting because presumably (s)he could 


receive the needed care in a less costly environment. 
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The levels of care concept work in the field of 
progressive patient care. Progressive patient care is an organizational 
esiutton which classifies patients according to their needs.” The cost 
of hospitalization is relative to the amount of care received. inthe 
course of an illness a patient may require several levels of cafe, For 
example, (s)he may be admitted to the intensive care unit, when 
stabilized move to a conventional unit, and then be transferred to a 
facility to Levels of care can be 
ee a a continuum from intensive care at one end of the continuum to 
no medical care nesged at ‘the other end, in view of medical and/or 
practical considerations, a esti ort need not necessarily require various 
bevels of care or changes in location during the course rs 


; ee A Concurrent Utilization Réview System 


The PSRO requires that the level of care required by a patient be 
assessed at prespecified time intervals, i.e., on admission and during 


the-pationt's hospitalization at points determined by the diagnosis or 
sialon ee sur ical treatment. The assumption is that these guidelines 
of the review process’ in meeting its 
objective of, minimizing misutilization of health care services 
facilities. The vanekbidiimiaes can increase substantially the time 

commit taint on the part of utilization review Committee members, A well 


organized concurrent utilization review system is necessary so as to 


ensure the efficient use of the committee members" time while not 


jeopardizing the institution's delegated PSRO status. 


UMI 


\ 


~ 
| 
‘ 


A utilization review system consists of the committee members, - 
physician (URMD) and other health care personnel, as well as 
administrative support staff. review system is: 
‘designed to conduct reviews the patient’ Ss stay. One 


type designed = to cybernetic control theory is | based on the 


principle that a it an organization, or an individual, etc. ) 
regulate itself if it. is provided feedback that it is functioning 
“incorrectly. To be effective it is cial that the feedback be 
explicit and timely, The system seeks to identify the patients who 
need subacute hospital care (Figure 1.1). and provides feedback to the 
att@liding physicians in hospitals about the actual and desired 
health facility resources. Ideally the feedback is then ei by the 
shysician and/or hospital to-regulate itself. 
The organisational of the utilization review systen 
cab vary. For -example, URMDs can review all the patients requiring 
review, make the Levenf6? care decisions, where necessary provide feed- 
| | batk to the attentive physicians .and perhaps to the hospital depart- | 
ments. In: fact, the system specifies URMD decide 
- when to notify the attending shesictan since PSROs are based on peer 


review. ? However, the use of URMDs to perform all the concurrent 


reviews required has several important drawbacks: (1) Physician time is 
a highly Lim; ted resource and is best spent providing direct patient care. 
It is poor use. of this resource to review cases that are > appropriately 

’ utilizing the hospital. (2) Concurrent reviews are not the top priority 


objective of URMDs given that ow are all practicing 


Under. the best conditions, a URMD allocates a limited sncusit of time 
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ACTUAL PATIENT\__ 
STATE 
CONCURRENT "ATTENDING 
UTILIZATION | PHYSICIAN 
SYSTEM 
PATIENT STATE ANO ACTUAL STATE ANDO 
| HEALTH SERVICES BEING | COMPARE TO DESIRED 


PROVIDED WITH LEVELS STATES 
OF CARE CRITERIA 


TO APPROPRIATE 
INDIVIOUAL OR 
DEPARTMENT REGARDING 
POTENTIAL MISUTILIZATION 


PATIEN 
AT DESIRED 
, STATE? 
DECISION MAKER 
. TAKES CORRECTIVE| ° 

| ACTION 


NO 


J. 
ORDER HEALTH CARE 
es" SERVICES TO 
ATTAIN OESIRED 
PATIENT STATE 
PATIENT RECEIVES y 
HEALTH CARE 4 
SERVICES 
% “DESIRED PATIENT STATE" is affected by information on alternative health care facilities 
FIGURE 1.1 A CONCURRENT UTILIZATION REVIEW SYSTEM 

= 


| 
| 
/steP 2\ /steP 3 \ 
PATIENT 
REQUIRES 
| ACUTE CARE 
| HOSPITAL LEVEL 
| OF CARE? 
| 
DISCHARGE 
> ES TO APPROPRIATE 
LEVEL OF CARE 
FACILITY? 
| 


each day for review, but given the Limitations, the timeliness of the 

feedback is compromised. (3) Recent findings indicate that vnen URMDs 
were forcibly involved in the decision process the feedback to 
physicians less effective. 

Case selection for utilization review can be a means for 
resolving’ the problems discussed above. Criteria can be used in 
selecting for the URMD,. thereby using her 

efficiently. Also, providing feedback corrective action, 
minimizes inappropriate utilization, '° hence, not jeopardizing the 


delegated Status of the committee. 


A monphysician reviewer, for example a Utilization Review Nurse ~ 
Coordinator, URNC, can be @mployed to select cases for review. This 


position is recognized by the PSRO program for assisting with the 
required paperwork and determining cases for URMD review. Various § =, 


criteria exist for determining which cases the URNC should identify for 
the URMD. Depending on the selection procedure chosen, the URNC can be 


minimally involved or can be involved through step 3 of the flowchart 
illustrated in Figure 1.1; having direct contact with attending 
physician and hospital departments. The choice of information to be 


used, criteria for case selection and organization of the utilization 


review system affect the effectiveness. and efficiency of the process. — 
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I.4 Case Selection in Utilization Review _ : 


‘ 


1.4.1 Presentation and Critique of Several Case Selector Methods 


PSROs are requiring: that Length of stay by diagnosis be the 
criterion for determining the day tor conducting the continued Stay 
review. With few. exceptions; — are requiring that continued stay 
reviews must be conducted by the 50th percentile length ot eee by 
diagnosis or surgical procedure and if the patient is considered appro- 
priate on that eee next review be conducted on the 7Sth percentile 
‘a of stay, and then the 90th percentile length of. stay. This 
assumes that t for many patients the 50th (or, 75th or soth) percentile 
Length of stay may. coincide with ‘the moment the level of care may be 
changing from acute hospital to another level. ‘It also precludes the 
of misutilized prior to the percentile, 


between the 50th and’ 75th percentiles, as well as between the 75th and 


90th. The efPicacy of this method has not been evaluated. 
Various strategies have ‘been developed for selecting patient 
_ charts to be reviewed retrospectively by the Utilization Review 
: Committee. The factor Common to most of the selection methods vas 
length’ of stay by diagnosis (e.g. nan Wolfe).’ ‘The underlying assumption 
don. these methods ane that patients whose length of stay fell outside 


the acceptable af limits for a ‘Gagnosis had a high 


bility ‘of being 1 ieecinialiah utilizers of an acute hospital. By 


cases in this. manner, physician time would not be wasted 
reviewing patient: charts who had not "misutilized" the acute hospital. 


wesiapriiend this assumption, did not perait the assessment of possible 
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misutilized days for "normal" lengths of stay patients. 


Additional variables such as age, Sex, and type of treatment - 
| 
could be used to specify differences among categories of patients. 


Although the categorization could make the selection process more sensi- 
- tive to differences between patients, "the use of a single criterion 

variable eich as length of stay may be insufficient to separate appro- 

and” inappropriate utilization with any degree of success", 
Emphasis was placed on length of stay in the reasonable belief 


‘that the great majority of hospital misutilization a found in the 
latter stage of hospital stays,.° Observations by Rosenfeld et. al... 

in their study of reasons, for prolonged aeaeee stay led them to con- 
clude a review data» that it would be unrealistic? to 


classify the S institutional requirements length 


of stay or chronicity of illness. No relationship was found between 


length of sine and the need for acute hospital Stay. Their findings a5) 


have been extended to shorter hospital stays by the work of Gertman, 


Bucher,’ and Zimmer. 8 They found that the majority of inappropriate 


days were “found in non-longstay cases, actually 65% of the inappro- 


priate days occurred in, cases wars ‘oe length of stay was below the 


w 


50th percentile of the matched PAS 
These results were then interpreted as showing that : 
the data indicate that there is no positive relation 
- ship between inappropriate days and length of stay 
deviations from statistical norms for disease 


Categories. This raises a serious question about the 

validity of using statistical length of stay measures 

as a’ basic audit or regulatory mechanism to control 

unnecessary hospital utilization. 7 = 
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In the Gertman and Bucher study the determinant of hospital level of 
care was the intern's response (the intern piapanathie for the patient's 
care) to a question requesting. the major reason for the patient's 
previous day in the hospital. ‘Restuccia and Holloway” used explicit 
‘criteria to arrive at a level of care decision and obtained results 
similar to those found by Gertman and pucher.”7 


These ‘various studies illustrate the inadequacy of this type of 


emphasis 


hich can lead to the attitude that if a patient has reached | 
the average length of stay that — should or must be ready to 
1 
leave the hospital of individual neceisi ties, 


McClain recognized this shortcoming and approached the problem 


by looking for additional criterion variables which would be useful in 
case selection. By using physician-developed, explicit criteria and 


non=-physicians applying the criteria, the cases selected for the URMDs 


» 


should in the aggregate have a higher probability of having misutilized 


days. The objectives of this utilization case selection program oan 
both statistical and screening based on pre= 
established medical criteria applied by non-physicians, 4720 py 

modelling the utilization x review decision process of physicians, he 
developed a decision aid in case selection for nen-physicians. The | 

model was developed for one diagnostic group (gail bladder and 
‘the fact that zt is diagnosis-dependent presents the problem of 

developing na models ‘for each diagnostic group, or at best, for 


wh each class of similar diagnostic groupings. 


The retrospective case selection methods described could. be . 


adapted eonGoncurrent réviews but here. are drawbacks to each. The 


‘ 
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first relies solely on Crate of ‘stay cut-offs. As already stated, it 


would miss. misutilized days in the "normal" lengths of stay ene The 
second method is diagnosis-specific thereby making it very cumbersome 


to develop and apply the criteria. 


1.4.2 Levels of Care Criteria in Case Selection for URMDs 


Explicit criteria and non-physician reviewers can be effective 


| | | 
in case selection for the URMDs.~° Given that the purpose of the con- 


‘tinued stay reviews is the identification of inappropriately located 


patients as a function of level of care needed, level of care criteria 
should be the criteria used by the URNC. (the non-physician reviewer). 
Levels. of care eviteria are diagnosis independent. The criteria 


are developed by the Utilization Review Committee and dpecity the types 


of. services which can only be provided ata hospital level of care. To- 


-arrive ata devel of care decision the URNC uses the criteria 2 angcrules 


for their application. The rules for applying the criteria have been 


extracted from the Medicare Hospital and Extended Care (Skilled iursing) 


Facility Manuals,” and translated into a level of care decision low~ 


sheet for the URNC (as illustrated by Figure 3.3)0 A detailed 


explanation of the decision process is discussed by Holloway et. a,” 


The appropriate level of care is a function of the medical 


services needed by the patient As well as the resources available in the 


& 


community. The community resources affect the generation of 
institution-specific criteria. The attending physician's current orders 
specify the kel vindins needed by the patient. For each day of 


hospital stay the patient can be characterized by a mediéal services 


b 
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appropriate level of care. 


quent reviews and analysis. 


14 


profile, e.g. the current orders. The medical services profile Changes > 
auring the course gation, Some services can be provided 
only . a hospital, whereas others may be provided in various health 
care settings, incliding hospitals. ‘The medical we profile is 


the primary information sourcé used by the URNC when deciding on the 


@ 


“The URNC and the decision are the selector method for 
the URMDs. This case Selection method provides an approximation of the 
decision to be made by the URMD. By the use of URMD-developed criteria, 
we in effect guiding the -URNC's decisions. A URMD first 


becomes involved in an individual decision when the URNC requires 


‘assistance in cases not evidently complying with the criteria. The 


4 


review process: decisions Pen Green domain of physicians (i.e. crite- 
ria specification), yet physician time is used efficiently because they 


only become directly involved by exception. : 


~ 


o¢ 


1.4.3 Computer-Aided Case Selection 


Computer technology can be and is being used to assist the URNC. 


For example, the system developed at the Laboratory of Computer Science, 


Massachusetts General Hospital, provides administrative support 


. necessary to maintain the patients' files, sdetermines automatically the 


length of stay review date based on-diagnosis and associated length of 

stay norms, and collects pertdnent data about each patient for subse- 


4 


_ A more sophisticated: computer supported patient classification 


_ system has been developed and is being used for utilization review at 


| | 
: 
| 
= 
| | 
; 
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Yale-New Haven Hospital. This utilization review system groups ° 


patients based on variables such as primary diagnosisy age, sex, type 


® 


of surgery, service, and type’ of secondary diagnosis. The variables 


and their associated values that define particular groups vary for each 


' group and were determined wd their medical and statistical significance. 


Lengths of stay norms for oath of these patient groups determine the i 


review dates. 

Seetens such as these assume that patients in the same Gamestic | 
‘category or patient medical services profiles 
that the 50th, are good ‘of devel | 


Computerized evaluation of an individual's medical services , 


~ profile should provide a better indication of a | Level of care change. 


“In a hospital with a computer-based medical record system, a model which 


approximates the URNC's decision wabisig. con screen each ‘patient every . 


day and select URNC review only those whose’ medical 


services profile indicate a significant pr hy. OF 


hospital utilization. ‘Computer-based administrative support systems: 


may be acceptable when the patient information is limited, that is, ' 


patient identification, age, diagnosis, etc. However, when more per- 


tinent information (i.e. medical services) to level of care assessment 


is available, predictive capability must be increased. 


& 
; 
= 
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t.9 ‘Development of a Computerized Level of Care Selector Model _ oY 


fie concent of level of care as a function of by 
the patient has been proposed. |” When 10 URNCs were presented with 
of current orders (patient tedical services profiles), they 
could reliably make level of care decisions." ‘Thus, the folloving 


may be inappropriately 


in an acute hospital are Characterized by a 


Hypothesis 1: Patients who 


medical services profile (i.e. current orders) 
which differs from that of patients who are 
probably appropriately located. 


Under this hypothesis, discrimination metHodology will be used 


to develop a mathematical model capable of differentiating the medical 


services profiles of one patient group, SELECT FOR REVIEW --- possible : Pai. 
inappropriate location, from the other, SELECT NOT FOR REVIEW --- ‘ 
probable appropriate location. In the assessment of a profile the 


value assymed by the model will be an index of the level of care needed 


by the patient. Henceforth, it will be referred to as Level of Care. 


Index, or L.0.C.I. 
Expert opinion and statistical methods will be used in the model 
development. Incorporating expert opinion should enhance the medical 


~ validity of the results obtained from the application of the statistical 


techniques. Medical experts will: define a set of variables which 


mi ene represent the medical services which can be requested in the patient 
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Then they will determine the utility of the presence of 
different Levels of these variables. The statistical technique will be 
used to empirically determine the weight or relative importance of the 

) optimally sticrtathating variables so that the sum of the weighted 
variables produces for selecting possible sub- ° 


acute hospital level of care patients. 


Hypothesis 2: Inappropriate ‘utilization more accurately 
predicted by a level of care index (L.0.C.I.): 


than by length of stay by diagnosis (L.0.S.) 


Although Hypothesis 1 may be accepted, it may predict as well as OP 
: length of stay percentile cut-off points. The two models will be 


evaluated to ascertain whether L.0,C.I. is a more efficacious predictor 


than L.O.5S. 
| Hypotiiesis 33 When non-computer based information is added, 
probable inappropriate location is more 
selector model than by a length of stay : 
| | calector model | 


= ~ 


The models under consideration are selecting patients for URNC 
& 


review; selecting patients whose services profiles were not sufficient 


to make a confident decision on required level of care. If the addition 


of non-computer based information frequently causes a change of the level 


UMI 


of care decision, then ‘the efficacy of each selector model could be 
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affected. 


When using computer-based information only, the URNC can 


classify patients as probably appropriately located or possibly inappro- 


and therefore seek further information (Hypothesis 


ce | When the URNC adds. further information, the patient groups are 


"APPROPRIATE" -—- probably appropriately located --- or "TNAPPROPRIATE" 


: probably inappropriately located. It is for those patients who are. 


considered probably inappropriately located that a feedback message is 


initiated to the individual(s) (e.g. URMD, attending MD, discharge 
plaming department, etc.) who can take corrective 
Hypothesis 3 seeks to evaluate — model's predictive said for the 
appropriately located or probabiy inappropriately located 


groups --- the confident. decision ‘of the URNC. 


I.6 Summary 


“The choice of patients stil has an impact on the- utilization 
review process. ‘The selection process siiould be identifying | : 
as many of the inappropriately located patients without: unnecessarily 
reviewing many “appropriately located patients. 
predicting when a patient should be. reviewed. Computer support: for 
| case selection has been primarily Medes amiakien and based on limited 
patient information, most often length of stay by diagnosis. A 


computerized model of- the. level of care decision process is proposed. 


A Mandap?” evaluation of the level of*care and length of stay 


selector models will be conducted. The measures of performance are: 


1) the selector sensitivity of each model --- the ability to identify the 
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inappropriately located patients ana: provable) >) the 
| specificity of each model the ability to identify ‘the 
appropriately located patients (possible and probable), 3) predictive 
‘sensitivity and specificity of each model --- the informational value 
of a model ' s predictions, i.e. the probability that the model' TY 
prediction is correct. : ‘ani discuss the theoretical 
approach,’ analysis, chosen for model development. 
Chapter III will describe the experiment designed to test the 
hypotheses posited. Chap ter IV presents a system for 
“medical services _— provides the ability to define pe of variables 
for ee level of care Aécision making. Results obtained and 
comparative evaluation are the sabia of Chapter V; and the 


implications of the: results are in 
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CHAPTER II DISCRIMINATION METHODOLOGY 


II.0O Introduction 


Deciding on the appropriate level of care for a patient can be 


viewed as” an al location* problem. Classification is the’ process of 


organizing ‘iin eee relationships the members of a heterogeneous 


population of objects, individuals, or diseases , whereas, discri- 


mination is a process or rulé which. enables one to allocate an 


individual to the correct population when the individual's membership 


ina is not known”, Various empirical 


have been used in medicine to develop systems for automated diagnosis, 


prognosis, ‘and to’ between altemative treatments. Discriminant 


analysis te one empirical approach which has been shown to be useful. 
Lachenbruch> provides an exhaustive survey of methods available for 


discrimination. A Multivariate Logistic Function (a transformation of 


a linear discriminant function) will be used. in this research to con- 


struct a discriminant function which separates acute hospital level of 


* Allocation, in its. general sense, refers to a procedure which places 
an individual or object into separate groupings. Therefore, it encom — 
reat classification, discrimination, assignment, categorization, etc. 
Due to the fact that. this dissertation involves various types of allo- 
cation problems, henceforth, allocation will refer to predictions of 

the selector models, and assignments will. refer to the decisions. 

arrived at by the decision makers. Thus, the selector model allocates 
whereas the decision maker assigns. Classification will be used for the 


procedure which organizes the medical services into classes of medical » 


services, 


‘ 
i 
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er 

of care from onsacute hospi tel leveof care patients: ‘The rationale 

for the use of this approach will be discussed in the context of com- : 
parable endeavors in health care end taking into account the nature of 

the data. A brief conceptual description of discriminant analysis will + 

be provided first, followed by a detailed explanation of the Lddistic 
Discrimination Method and its relatioriship to the Bayes' Theorem | 


approach to discrimination. 


-II.1 Basic Concepts of Discriminant Analysis | 


The goal of discriminant analysis is to allocate. an observation 


X, where X is a vector of k x 1 predictor variables (X,, 

one of two or nore distinct groups. The group membership of an 
observation is unknown alidcation group is based on the 
value of Xe Although in’ some ismmaiaheein the distribution of X in each 
group is Known, in most problems the knowledge of the éistridution of x 


is determined by a relatively small. sample from the populafion being 


studied. This distribution of X, whether known or estimated, is used 
{ar Different methods for estimating the. 
Aiurpnation of X will be described in this chapter. 
The strength of any decision rule or discrimination method is 
its accuracy or specificity. Therefore a critical chenietios of discri- 
- minant analysis is por the error rate (or, misclassification 


rate) when allocating an observation to a group. Since the function, ..— 


used to discriminate is developed from data on group samples, the 


assumption that the initial data are correctly classified becomes vital 
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to the accurate development of the discriminant function, That oi in 


defining groups, there exist one or more variables that allow us < 


establish the groups. 2 The allocation of an individual observation is 


based on the degree of relatedness of the variables to each group. It 


is desirable to find a subset “of the variables which can best allocate 


an observation to the correct group. 


The discriminant function is a means of combining the information 


from predictor variables in order to discriminate, as well as possible, 


between individual observations. A linear combination or a transformed 


Linear Combination of the variables is often used as a SeScyaEnans 


function. An optimum weight is determined for each variable used in the’ | 
function. The method used for the weights depends upon 
: 
rule for optimization; C.Je; minimizing the total probability of mis- 


classification (Maximum Likelihood) or minimizing the maximum probabi- 


lity of misclassification in any one group (Minimax Rule). 
A conceptual description ‘of a discriminant function is as. follows: 
Let 4 and be two mutually exclusive and 


"groups to which an observation x allocated 


A linear discriminant function with 2 » variesbles is: 


~ 


tiere a(X) is the value of the discriminant function 
for observation X 
Xs X, are the predictor variables that take on 
values for each observation x 


Boy Bis Bo are the calculated weights 


| 
d(X) = Bo + Bix) 
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values) of a(x) for Group. 1 and naman 2. 
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Ir d(X) =0 is the ‘optimal decision boundary, then the 

| following classification rule may be used: : 


if a(X) 20 then assign X to 0G, 


if. d(X) < 0 ‘then assign X to G, 


| | 
This rule discriminates between individuals that belong to G, 


and Gye Figure 2.1 represents the set of all possible combinations’ of 


x, and Kye The point (X,, x 9) for each member of Group 1 is represented 


by Gas. and ‘similarly for Group 2 by Gye st can, be seen from Figure 2e Be 


that the two groups tend to” mort different regions of the wes, with 


‘some overlap. The amount of over lap affects the misclassification rate. 
Figure 2. 2 represents che density function (ai stribution of the 

‘The use ae a discriminant function | can be viewed as a test of 


the following hypotheses: 


~ Hypothesis 1: Observation X is a member of Group 1. 


, Hypothesis 2: ‘Observation X is a member of Group 2 
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| Theorem is limited to the case of two groups, G,’ and G 
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The possible outcomes of the classification procedure are: : | J 
(i) Accept H, when H, is true (True Positive) 
(ii) Accept H, when H, is true (False Negative) 
(iii) Accept H, when H, ‘is true (False Positive) 
es (iv) Accept H, when H, is true (True Negative) 
In Figure 2,2 the False Negatives are represented by the lined — 
area and the False Positives are represented by the dotted area. The 


goal of the discriminant function is to minimize these two areas. 


II.2 Bayes' Theorem in Discrimination 


piven Theorem is one approach which can be used for discri- 


mination purposes. If an observation P, of 


belaugiog to G, and a posterior probability Po of belonging to Sg ae 


the Bayesion approach would allocate the observation to G, if Py ? Por 


and to G, This the. expected probability 


of misclassification. | 


". For the purpose of simplicity, the following discusSion of Bayes’ | 


Bayes’ Theorem 


states that the that an observation X belongs to group G, is 


2 
a 
f 
a 
4 


the. probability is calculated for G, and G, 
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given by: 


"Where Pr(Gj) the prior probability of 


Pr(x|Gj) is the ‘conditional probability or likelihood of 


x given 


Pr(G;|X) the posterior probebility of Gy given xX 


| Assuming knowledge of the prior probebilities.aid the likelihoods, 


The 


classification rule is then used to allocate observation x to a group 


G.: 
Let Bayes'.Odds be equal to the ratio ‘ 
: Now, if Bayes! Odds 2 1 then allocate observation x to Gy 
if Bayes’ Odds <.1 then allocate observation X to Gp. 


2 


The prior prqbability can be calculated from the. sample or one 


can assume equal priors and assign a probability of 0.5 to each. 


minimax solution is achieved by exsiming priors.” Te the 


(the variables which make up the. vector X) are binary and assumed 


‘ 
é 
a 
‘ 
| 

‘ 


; puter-assisted diagnosis (see for example scented and Lusted 


‘independent, then R{X|G,) can be calculated by 
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Pr(X|G,) Terex. (Independence Model) 


Bayes' Theorem has been proposed as a theoretical basis for com- 


wits 


its usefulness in | classification is based on a universe of Roser 


mutually exclusive, exhaustive and infallibly determined groups too 


diagnoses). Efforts in the field of computer-assisted decision making 
usually have dealt with a situation where there is a small number of 
possible disease groups to-which an individual can be assigned. Gorry 


et. al.” vised Bayes? Theorem to sequentially update the probability. 


that an individual presenting with the symp toms of acute oliguric renal 


failure had one of fourteen possible diagnoses. As more information was 


obtained about the patient, the probability that the patient had a given 
disease was calculated. Medical expert opinion was used to estimate a 


priori probabilities of each disease group for an individual with the — 


presenting problem and the conditional probabilities for a symptom or 


sign in each of the disease groups. Ninety per cent of the time the 
program agreed with experienced nephrologists. | Although their obtained 


correct classification \rate was reasonably high, the authors identified 


_problems which needed further resolution before "computer consultant" 


systems would be useful. One issue they raised is that of symptom non= 


independence, 


# 


rrequently the assumption of independence of symp toms ls made 


when applying gases? Slecren in medical decision making even as it is 


» 
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2 recognized that this assumption is inaccurate. Small data samples have | 


made it difficult to determine joint probability distributions. The . 


* accuracy of subjective estimates oF joint probability distributions 


would be However, confidence. in. Theorem, as well 
as the difficulties. in obtaining joint probabilities have minimized the 
criticisms of this assumption, 

‘The consequences of the independence assumption were investigated 
by Norusis ena Clinicians incorporate eymp tom inter- 


dependencies in their diagnostic thought processes, yet most mathematical 


models tend to disregard or minimally utilize the information present in 


correlated observations. Symptom interaction is a serious problem in: i 
statistical approaches to Classification. Elashoff, et. al.’ studied 
the sebaehan of 2 out of n dichotomous items ina classification problén. 
They found that discrimination could be increased by choosing positively 
correlated items and decr’eased by choosing negatively corpeiated ones. 
This indicates that trying to use oy. uncorrelated variables may have 

a detrimental effect on the classification 

Theoretically, the full multinomial model (all possible outcomes) | 
4s the optimum théoretical distribution, however, the model's 
performance is seriously hampered by small samples. Por models of 
moderately separable groups, Norusis and Jacquez found that including 


second-order interactions, in the model substantially increased the 


rach of the results over the full independence approach many 


uUSe. 


For the problem studied. in this dissertation, (a) the groups are 


net easily separable, (b) interactions exist in. the minds of URNCs ‘and 


4 
i 
i 
i 
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URMDs when using the avai lable information in making a level of care 
decision? , and (¢) the data pry in decision making has been -dichoto- 
mized. é 

addition the independence and the: full multinomial models, 

a | variety of neatiotiens models exist which can be used. to estimate 

multivariate binary ‘distributions. ‘The linear discriminant 
cal approximation which incorporates second-order correlations. 
Al though binary data do not satisty some of the assumptions of the 
linear discriminant function, the robustness of the method to deviations. 
from both normality and equality of covari®hce makes its use 

plausible. in this allocation problem, The fact that it incorporates 
second-order correlations and withstands deviations fram ree 
assumptions, makes the linear discriminant function the logical method — 
to be used. The next show how the linear ca scrimsnant 
function is used to calculate posteriar probabilities and its relation- 


ship to Bayes’ Theorem. 


11.3 Logistic Model in Discrimination 


The Multivariate Logistic. Function is calculated by the use of a 
eentigienes transformation of ‘the linear discriminant function. It 
provides a means to allocate an observation to a group, as well as an 
estimate of cartainty with which the observation may be allocated. 


estimate of certainty is the posterior probability, and is calculated 


without the direct calculation of the likelihood or conditional 


: probability of X in each ‘group. 
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From linear regression we can assume Y is linearly 


related to a set of predictor variables, X 17 “Thus, 


Y = Bo + B1X1 + Boxe 
erry are the predictor variables 


Bi, Bo, B the weights associated with each 
variable 
In discriminant analysis, Bayes' Odds is the value one is trying 
to calculate. In equation (1) above, the coefficients (the Betas) are 
estimated by the maximum likelihood method!5 and ‘a value 
Calculated. (The linear used for stepwise 
selection estimates - the Betas by the use of the minimax rule. ) Since 
_ the goal is to allocate an observation to. a group, one needs to — 
which group Y is “most Likely ‘to belong to... The logistic function trans-— 


“forms the non-nortalized linear discriminant function into the 


interval, which is actually the probability. of allocation into Group 1° 


(posterior probability). 


The Logit of where p is the posterior probability 


Pr(G, x), is defined ast) 


s+ 


variable 
p 
| Y= ln 
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Now using this transformation and equation (1) we obtain 


l+e 1+e 


which is called the multivariate logistic function e 


The classification rule is: 


O then allocate X:to 


iV 


The rationale for the use of the Logit is related ‘to ‘the lack: of 
knowledge about the likelihood. of X as well as the desire to know the 
certainty with which X has been allocated to one group or the other. 
The preticted values are necessarily in the range [9.3] and reflect 
the classification certainty factor, which is the same as the posterior | 
probability obtained from Bayes’ Theorén. The following steps . | 


illustrate how the results obtained fron the logistic ae are 


similar to those obtained from. Theorem in the case of a aml 


normal in each group. 
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Let I = prior probability of G, 


. 


£, (x) = multivariate distribution of X in G, (the likelihood) 
£,(X). distribution of X in G, (the likelihood) 


posterior probability that X belongs to G, 3 


(equivalent to Pr(G, |X) 


The probability that a given observation X belongs to G., is 


equal to 10 


#1(X). 


1 


If £, (Xx) and £,,(X) are multivariate normal, then 


(1-11) (X) 


- (Bo * 
#1 (X) 


_ where 8, are the coefficients of the logistic discriminant 
function estimated by maximum likelihood. 


Although the method is optimal if the observations are multi- 


variate normal, the posterior probability can be calculated for X by 


approximating ‘the distributions of £, (x) and £,(X) even if they are not 


multivariate normal, When the distributions of X given G (i.e. f, (X)) 


a 
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are known or can be estimated subjectively, the Bayesian approach pro- 


an wides a good. discrimination rule. It has Jats been shown how the logit 


transformation ailqvs us to arrive at the same allocation rule by the 
use of approximations or enpirically-determined estimates. 


Armitage, for example, used a logit ‘transformation to predict the 


of with advanced breast cancer. ‘He ‘waited to 


determine ‘the predictive, variables for’ remission of the disease so that 
iis a new patient would be seen, the decision for type (if any) of 
ablation would be a function of the — of. discriminating variables 


with in predicting prognosis. He found that sblation, 


mastectomy, urinary steroids, the log of" time (either from time first 


seen or time from mastectomy to ablation), and the interactions of these 
variabies, are good predictor variables. HiS results of the effect: on 
prognosis of the choice of ablative procedure eécarés well with the 
comparison provided the randomized trial of Atkins et. a. 

with a Clear indication in Saee of one type of ablation in the prognosis 


1 
for the first. year. after ablation. 18,1920 


II .4 Logistic Discrimination in Utilization Review Selector Method 


In the research undertaken in this itl the predictor 


| variables, the X. ;» are Clusters of medical services, The medical 


services’ were subjectively classified into the X. “by utilization review | 


decision makers (procedure described in Chapter IV). In view of the 


oreceding discussion, an empirical approach seems appropriate for 


deriving the weights or probabilities associated with the Xie Actually, 


deriving the weights from a representative population via a methodology 


> 
‘ 
¥ 
34 
F 
> 

| 

4 


35 


such as discriminant analysis "appears to be the only proven approach 

where there are interaction terms in an additive model" .° Therefore, 
the empirical approach, a logistic discriminant. function, has been 


choses for this allocation 
An automated selector method which- approximates expert level of 


care assignment will - allocate a ‘patient to 


Ge = = SELECT NOT FOR REVIEW based on the service clusters present : in the 


= SELECT FOR REVIEW or to 


patient's medical services profile. 


If the same population is used for both estimating. the weights 
al eee the proportion of misclassification, the poor quality of 
the estimates may: not be obvious. Therefore, it is desirable to use the 
estimated coefficients to classify a a new set of enewianian end then 
the proportion of individuals misclassieréd Using ‘this pro- 
cedure poor performance of the discriminant function may be obvious in 7 


a much higher misclassification rate. 


The sample population was partitioned into an estimation set, 


used to estimate the coefficients and an evaluation set, used to 


the discriminant The proportion of observations 
which had been assigned to the SELECT FOR ee and SELECT NOT FOR 
REVIEW groups, was approximately the same for the estimation and 

sete. The initial pattern of variables which could be-used 


ina discriminant function most likely exceeds the number of variables 


which are needed for discrimination purposes. That is, there are some, 
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included in Chapter III. 


and perhaps many variables in the initial pattern with little or no 


discriminatory power. A procedure for selecting the variables that 


discriminate best is applied to the-total variables, In- 
this study, | a stepwise procedure was: chosen, The variable which 


discriminated best between the two groups was chosen first. ‘The next 


variable ‘chosen was the one which 1 contributed most to further discrimi- 
nation. This continued none of the renaining variables 
contributed: substantially to "discriminating between the groups. 


Additional details about the procedure fos variable selection are 


“a 


II.5 Summary - 


_Bayes' Theorem, as well as linear discriminant analysis, are 
methods uséd in computer-aided decision processes. The problems and 
advantages associated with each method have been presented. A logit 


transformation of a linear discriminant function permits the realization | 


of some of the advantages of each method, and therefore seemed the 


‘logical choice for the computer-aided allocation problem under 


consideration, 


Empirical, classification methodology is based on the assumption 


that individuals in the population from which the parameters will be 


estimated have been correctly classified. In situations where an 


absolute "true"—classification is not available, expert generated 
& 


criteria and rules ensure reliable classification of observations. The - 


remainder of this dissertation describes an experiment that creates a 
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data base of patient days which have been reliably assigned into one of 


two groups,. SELECT FOR REVIEW and SELECT NOT FOR REVIEW, details the -. 


information used in the assignments, and uses the data to develop a al 


discriminant function for utilization review selection. 
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CHAPTER III METHODOLOGY 


Introduction 


To determine ‘the characteristics of a population which Permit 


a 


members. ofthat population to be ‘separated into subgroups, one must 


study a sample of the population and identify those characteristics 


which define the subgroups. An experiment was designed to accumulate 


pertinent data on a sample of hospitalized patients to provide the 


information needed to test the hypotheses under consideration. This 


Chapter. consists of the following four sections: (1) the experimental 


design, including tests ‘of reliability of decision making ang predictive 


validity of the decision process being modelled, (2) the data base 


construction, (3) the model development by the application of the 
Loststis Discrimination Method discussed in the previous chapter, and, 
(4) measures to evaluate oe selector models, a Level of Care Index 
(1.0.C.1.) and the currently Length of Stay by Diagnosis 


III.1 Experimental Design 


An objective of this study was to develop a decision model to 
automatically differentiate patients who are appropriately located in an 
acute hospital from those who may not be. To implement the approach 


chosen, it was necessary to collect a data base of hospitalized patients, 


to record the medical services ordered for those patients, and to 
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obtain an expert judgment of the level of care facility a patient needed 
ona given day. 
| 
The study was contacted at a 164 bed, non-profit, tax-supported, 


short-term general. communi ty hospital serving patients under the care of 


their personal physicians. the hospital was chosen as ‘the site for. this 


project because it a computer-based total hospital information 


system. The system, Technicon MIS, has been in use in the hospital since 


‘1971. ‘Hospital staff and physicians have had ample time to acclimate to | 


the system; this factor is important to providing confidence in the . 


_ information obtained from the system. 


MIS isa real-time, computer-based systen that nurses, physicians, 
other’ health care hospital administrators interact with 


in the delivery of health care to. patients. iesievoue: of a computer 


aided selector method is ‘@ependent on patient ‘information existing in 


computer readable form; A major portion of the patient's medical 


record is stored in the computer's data base during the patient's 


- hospitalization. Physicians enter medical orders and request results 


via Video Matrix Terminals installed at each nursing station, and at all 


other departments (@.9., clinical laboratory; radiology, etc.). In 


addition, a patient chart paper record of patient medical information 


- is maintained at the nursing station and subsequent to discharge filed 


in the medical records Separteent.: The chart, which reflects the 


Computer medical record, is composed of periodic reper. eetaced by MIS, 
and hand written entries by attending physicians and nursing staff. 
Various reports are printed, among which are current medical orders, 


laboratory results, cumulative laboratory results, x-ray reports, 
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medications to be administered, etc. The Utilization Review Nurse 


Coordinators use MIS for conducting end ‘continued stay 
reviews, 
we — For each patient the computer medical record documents all 
| medical orders ana examination results fromathe time of “an: 
evaluation of the completeness pee accuracy of services rendered has 
been conducted by an independent research institution.. They’ found: 
that the rate of completeness and accuracy of patient information ts 
significantly higher than prior to the installation of the ivaten, 
Therefore, in this research it has been assumed that the computer 
medical record documents all the hospital resources the patient needs on . 
Given day. | 
It is important to assuniption since the computer 
“medical recor'a ‘is a major source of information used in making a level 
a care decision about a patient. Thus, if the completeness and 
accuracy of the data are inadequate or inconsistent, the predictive 
validity of the decision model would be compromised. 


> : 


TII.1.1 Sampling Procedure 


Patients from the following specialty units were eligible for 


inclusion in the study sample: General Medical, General Surgical, | 
Gynecology, Orthopedics, and Urology/EENT. The high intensive units 
‘(Intensive Care Unit, Transitory Care Unit, Coronary Care Unit), 


Psychiatry, and Pediatrics were not included in the population sampled. 


It was assumed that an individual in a high intensive unit required 
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at_ least a hospital level of care. Psychiatric and pediatric medical 
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services differ from those for the rest of the hospitalized population 
_and thus it: was felt that the proposed model would not hold. Hence, 
The sampling period started in June 1975 and continued for a 
period of six months. Each day a sample of patients from each of the 
units was chosen, An patient day is the unit 
of ébservations thus, d patient may appear in the sample more than once. 
Weekend days were included. — each Unit, I; N, patients were — | 
randomly selected for review, where N, is proportional to the census on 
the unit. In addition, atuty sample patients who 440 riot Medicare 
acute care hospital level of care criteria on a given day were peviewsd | 
"on each absaquant day until they either a) returned to an acute care 
| hospital level of care need, or b) were discharged. Previous studies 
have — that using the same Medicare level of eure criteria 10-20% 


of patient days are considered ‘inappropriate.~ They also demonstrated . 


83% of inappropriate by at least one additional 
inappropriate day. For the empirical method 
small proportion of cases in one group creates 4ifficulties in model 
bivsicunent. By including patient days subsequent to the "inappro- 
the sample was biased to a bere. 
centage of "inappropriate" days as well as carnitine follow up on 


III.1.2 Patient Data for Decision Making 


An abstracted form of the computer medical record was the primary 


~~ 


information source. For each observation two computer generated reports 
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were printed. “Report. 1, Part A is a list of all current orders which 


will be used as the predictor variables. in the decision model. besor't T, 


3 


Part B lists demographic and diagnostic information and outdated medical | 


orders (sen Figure 3.1A and 3. 1B). Report I was printed for each 


patient day reviewed. Report II is a discharge summary for study 
patients od is printed within two days subsequent to discharge (see 
Figure 3.2). 

Report I was used for decision making (to be explained in the 


next section). Report II was not used for level of care decision making 


but was needed eds the > Length of stay data to evaluate L.0.C.1I. versus 


: L.O.S. as selector models. In addition, it was oe to know the 
discharge diagnosis as. well as the admitting diagnosis and the diagnosis 


current on the day of review. The reference tables used in assigning 


L.0.S. review dates are based on discharge diagnostic information which 


is not always available, especially in the computer medical. record, 


until the date of or sometimes after discharge. The completeness of 


Report II depends on the physician's use of the system to discharge the 


patient. Approximately 50% of the enter 


information . (discharge diagnoses, and follow up waesecbacteess such as 


, where patient is discharged to) via the video matrix terminal. Other 


physicians include this information in cae dictated hia summary, 


in which case the data was cient from the patient's Chart. 


Determination 


4 


The decision process for a level of care assignment consisted of 
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multiple stages of gathering. Using the flowsheet 


(Figure 36 3) a level of care assignment was made with a new sheet used 


for each stage. At. each stage, the URNC assigned the. patient to one of 


the following levels of care: 


Acute Hospital. 
Ideal Skilled Nursing 
Existing Skilled Nursing Facility 
Home Health 

Home 


_4 


The sequence was terminated when the —_ felt confident of the accuracy- 
of her  Suachiene’ If she per the level of care. assignment she 
indicated which item of information scebiced the change. Figure 3.4 
isa Plow diagram for sequential information acquisition. The sequence 
had to be maintained in the information gathering process. ‘Since the 


accuracy of a level of care index would see in part, a reflection of the 


_ information used in its development, it was important to know the 


information used for decision ishing, and whether the information source 


A 


was 
' In summary, for each observation the following data was collected: 


a” Report Ly Part A | Current Orders 


Part B - Demographic and Diagnostic Information, 
All. Outdated Orders ‘since Admission 
(2) Report II - Discharge Report 
(3) Judgment URNC Level of Care for each 
stage of information 


The decision model is to approximate the URNC's decision making 


process in level of care assignment. | An underlying assumption is the 
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I. Availability of Skilled Nursing Service at All Times 


_ Definition: A skilled. service is one which Cannot be 
safely and adequately performed by the 
average, rational, nonmedical person 
without direct supervision of trained 
medical or Daramedical personnel 
{including observation and instruction). 


I.A. Observation. Does the unstabilizea condition of 
the patient require "the skills of a nurse to 
detect and evaluate (i.e. observe) the patient's 
need" for possible modification of treatment or 
institution of medical . 


I. Be Direct Service. Does the I.B. Direct Service. Does the 
patient require direct patient require direct . 
skilled nursing services =: skilled nursing services 
(excluding observation) _ (excluding observation) 
every day? every day? 

3 circle one: 


; -I.C. Patient's Condition. In view of the 
Patient's condition, are the range 
(number. of different skilled services) 

and intensity (frequency or duration) 

of all:skilled services (e.g., exten- 

sive diagnostic tests) furnished such 
that they cannot be performed outsgde 

an institution? | Y 


NO 


{De Is the patient terminal? 


YES 
NO. Is the patient! condition 
unstable? 
NO YES 
Not Acute Hospital Acute Hospital Go to 


(Assign 1 or 0) (Assign 4) ° 


MEDICARE. LEVELS OF CARE-CRITERIA 
Medical and Surgical Patients 


LeAels 


I.A.2. 


If yes, describe 
the unstable 
condition(s): 


If yes, what about. 
the patient is the 
nurse observing to 
keep tabs on the. 

unstable condition? 


If yes, when fhe 


describe the 
skilled service 


_. (not necessarily 


I.B.1. 


nursing) that would 
be required: 


Give the most 
important service(s) 
rendered and give 
frequency: 


Frequency: 


If yes, what is the 
patient's condition 
such that the service 
cannot be performed 
on an outpatient 
basis? | 


I.C.2. What service(s) would 


be rendered if the 


patient's condition 


I.D.1. 


worsened? 


If yes, describe the 
unstable condition: 


FIGURE 3.3 DECISION FLOWSHEET FOR LEVEL OF CARE DETERMINATION 


4 
NO 
unstable 
manifests itsel 
YES 
| 


II. Acute Hospital Medical Services | MEDICARE LEVELS OF CARE CRITERIA 


II.A. Having established that the patient requires II.A.1. If yes, why is this . 


availability of skilled nursing services at : : constant availability 


all times (or requires broad and intense of a physician, required? > 
skilled services in an institution), does the : 

patient today require the constant availability «| 


NO YES ~ 
II.B. Does the patient today : II.B..Does the patient today § I1.B.1. Give the most, important 
ores 3 require the constant require the constant service(s) rendered (if 
‘availability of medical availability of medical Rehab Services are the 
: services provided by an services provided by an only services, then go to 
acute hospital and not acute hospital and not © the REHAB WORKSHEET II): 
ee provided by a Skilled provided by a Skilled 
Nursing Facility (SNF)? Nursing Facility (SNF)? 
NO YES circle one: 


NO _ YES 


Is the patient terminal? Acute Hospital Acute Hospital 


(Assign 4), (Assi 4) 
“Is the II.B.2. If yes, describe the 
condition unstable? unstable condition(s): 
NO “YES 


Skilled-Nursing Facility Acute Hospital 
_* (Assign 3 or 2) (Assign 4), 


FIGURE 3.3 (continued) DECISION FLOWSHEET FOR LEVEL OF CARE SeTERENATION ats : 
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STAGE 1 
(computer-based) 


REPORT I PART. A 
__ current 


INFORMATION 


Are you patient 
oy needs acute hospital level of.care [ 


YES #\no, why? 


Stop, go to 
next patient 


Demographic,Oiagnostic. 
Outgatec Orders Since 
Are you confident patient . 
needs acute hospital jevel of care 
"YES NO If no, giver 
7 present info, 
STAGE 2 Circle Orders 7 what level of 
INFORMATION That Provided care does patient 
(computer-based) | Information far need 
‘Stop, goto . Are you conficent 
next patient oF your Gecision? 
y Are you configent patient 
PP ss acute hospital level of care (4)? 
YES what level of 
care does patient 
Do nt need 
ti 3) Information ds ds 0° 
Arrive at | Are you confident 
Decision of your decision? 
; next patient 
PATIENT 
“Are you configent patient: 
peeds hospital level of 
STAGE 4 | NO Wheat level of care 
INFORMATION = does patient neec 
Document 5, 2 i. 9 
Used for Are you confident 
Decision, of your decision? 
Stop, go to YES | 
next patient 
[HEALTH PERSONNEL CONTACTED 
; Are you confident patient 
> 
STAGE 5 needs ecute hospitel level of cere (4)? | 
Document who was contacted and | 
YES 3 NO. Patient should 
be located at 
next patient Document Actions Taken 


Document Berriers 
to Appropriate Location 


. FIGURE 3.4 SEQUENTIAL INFORMATION ACQUISITION FOR LEVEL OF CARE DETERMINATION . 
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49 
“accuracy of the URNC's judgment in making the assignment. This 
must: be verified prior. to the. model The 
presence of random errors.in making assignments undermines of: 
lawfulness that an anpicicn model can uncover. therefore, the reliabi- 3 
‘lity and. validity of the judgments must be assessed. 

‘The: URNC can. be said to be reliable in ‘level of care assignnent 
af she is conetiscune in her decision making. Thus, consistency or | 
of results isa measure of reliability. Mutual consi- 
stency between different individuals of opinion) is 
referred to as inter-reviewer carta and is one method for 
determining reliability. is a necessary condition for 
assessing predictive validity and must. therefore be established, The 

{assessment of predictive validity of the URNC's judgments will. be | 
discussed in the following section. 

The decision flowsheet abstracted from the Medicare Manual 
(Figure 3.3) specifies both the criteria and rules for the application . 
of the criteria and was used to intra and | 
reliability. To measure inter“nurse-reviever reliability, another nurse, 
henceforth ieee to as ‘Nurse es was specially trained in levels 
of care criteria and their eer red She had previous experience in 
making levels of care decisions” ‘but did not work at the hospital. 


' For each observation she used Stage 1,2,3- information, that is, all the 


information that may be available in the computer medical record, to 
fiake an assignment ‘to the. SELECT FOR Review or SELECT NOT FOR REVIEW 


group. The computerized patient sie wet is to determine which 


patients should be selected for review by the URNC. Thus, it is 
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of agrednent vas calculated. 


30 
sufficient that this is the decision to be approximated. To be Capable i 
of comparing the judgments the and the Nurse Coder, the 
iebeutbonn the URNC had assigned to non-acute hospital level of care 
(any judgment less than 4) from the use of Stage 1,2,3 information were 
reassigned to the SELECT FOR REVIEW group , Gae Those observations 
assigned to’ acute tat level of care using Stage 1 ,»3 information 


were reassigned to the SELECT NOT FOR REVIEW group, Ge. (see Figure 3.5). 


8 


An inter-nurse-reviewer reliability intes as measured by. a coefficient: 


‘ITI.1.5 Predictive Validity of the Nurse Judgments 


Validity tefers to the extent that the instrument being developed’ 


is useful--that is, if it does what it is intended to do. A decision 


model is valid if it satisfactorily approximates the decision process. 


The type of validity at issue in this problem is predictive validity. 


Predictive validity is of concern when the purpose is to use the 


instrument to estimate an important form of behavior. The, behavior is 


> 


the criterion by the validity of the instrument is 


As already stated, the URNC is (a screen of selector method for 
the URMD. An underlying assumption of Utilization Review is that 
ultimately only a. physician is qualified to review another physician. 
because their experience and SPaBING provide them with a —_-s 
body of knowledge ‘to make decisions. A URNC has a "limited" aspect of 
the knowledge base and may therefore be 


estimating what a URMD would decide with the use of a more "complete" 


knowledge base of medicine. The URMD's decision is the criterion by 
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which the judgments are evaluated.’ Both have equal information 
for. the specific patient buts differing general knowledge about the 
patient's problems. How ell foes the vo differentiate the patients 
the decides need acute hospital lee oF care from those the 
decides rieed more detailed review? 
| A randomly selected sample of 99 observations was used to : 
‘evaluate the predictive validity of the murse judgments. Two physicians, | 
| with extensive training and experience in utilization review ‘ae levels 
ok care decision naking, made. a level of care judgment for each 
observation in this sample by using only computer-based (Stage = 3) ee 
information. Inter-physician-revi ewer reliability was calculated as well 
as the degree of predictive validity ‘of the murse judgments. The 
predictive validity of a model (inthis case the URNC is a model for the 
‘ym0) Consists of calculating the. coefficient of agreenent between the 
nurse and reviewers. higher the value of the. coefficient 
of agreement, the ince the predictive validity of the nurse‘s 


making. 


III.1.6 An Index of Agreement for Reliability and Validity Assessment 


An often used measure of reliability is the proportion of perfect 
agreement between the bexiauete, The Chi-square test and the contingency 
coefficient, C, 5 are two other popular measures. Simple proportion of 
agreement does not take into ‘account the 1a bios of agreement which could 


be expected by chance alone, Chi-square and the contingency Coefficient’ 


C, measure association and not agrecnent per se. A valid measure of 


agreement should incorporate a correction for the degree of chance 
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of agreement..’ 


agreement, it should measure agreement and not just association, and 
should be amenable to a test of etatistical significance of the degree 


Kappa, a coefficient of ——— for nominal 


scales, provides these factors«that is (1) it measures agreement 


corrected tor: ‘that which is expected purely by chance, (3) it is scaled 
bron to where negative values indicate worse than chance agree- 


ment, 0 indicates exactly chance and positive values indicate better 


than chance, and. (3) it has a well defined standard error which permits 


statistical assessment of the significance of the observed degree of 
| 6 
agreement.. 
The following table is presented to illustrate the components 
used to calculate Kappa and the approximation of the standard error of 


Kappa. 
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DECISION MAKER A 
Po, 11 Po, 12 Py. 
| 
| Pe, 11 Pe, 12 
¥ | 
Po, 21 Po, 22 Pa, | 
| | 
O 
Pe, 22 


Where Po. 


Pe,ij and is 
in 


em the observed proportion in cell ij 


the expected proportion 
cell ij 


Kappa is simply the proportion of agreement after chance 


agreement is removed and it is calculated by the following formula: 


where * 4; i=j and 


43 for i= =j and 


is the observed proportion 
of agreement 

is the expected proportion 
‘of agreement 


0,1) 

Me 

l- 
Pe | 


a 


An approximation to the standard error of K is as follows and 
has been defined as the conditional standard error by Bishop et. a 


(the standard error of kK, 6, is the square root of the-Var(K)). 


is 


If N (the sample size) is large (N 2 100), which is the case in 


this study, the sampling distribution of K will approximate normality SO 
that confidence limits can be set in the usual way: _ 
95% Confidence Limits = Kk *.1,96 OK 


99% Confidence Limits = k * 2,58 0, 


By evaluating the normal curve deviate, z, one can test the 


significance of the difference between two independent K*ss 


Ky - Kp 


| 
To test an obtained K for significance, that is, testing whether 
m there is any agreement between the decision makers; or equivalently, 
testing the null hypothesis that K comes from a population of units for 


which the population K, ee Calculated as follows: 
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The approximation to the standard error for the case of no agreement is 


as follows:® 


Thus, Kappa permits the assessment of reliability and validity 


based on the degree of agreement. 


III.2 Data Base 


The total amount of information necessary to make decisions 
exceeded the quantity of data needed to test the hypotheses under 
consideration. Therefore, the data base created to develop and evaluate 


the L.O.C.I. and L.0O.S..models consisted of diagnosis specific length 


"OF stay data, level of care judgment(s), and a coded version of the 


oO 


medical services profile (Report I Part A). : : 


III.2.1 Length of Stay Data 


The L.0.S. selector model is an adaptation of the utilization 


review admission and continued stay diréctives of the San Francisco Peer 


Review Organization. They in turn, have incorporated the American 


Medical Association and, PSRO recommendations for conducting utilization 


review.’ 
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An initial length of stay assignment for the current® diagnosis 
determines the date of the first continued stay’ review. Percentile 


Laegeh of stay guidelines are diagnosis and age specific as defined by: 


_ the Professional Activity Study (P. me Se ) for Western Region Hospitals. ~ 


visas are to be reviewed when their length of stay is equal to. 


the 50th percentile pesca of Stay for their respective diagnosis group. 


if determined appropriate on that day, the 75th percentile length of | 


4 


stay is the day of the ree eer review, and then the 90th percentile 


length of stay." 


The following rules are wie in ascertaining the "correct" length 


of stay assignments for the L.0.S. model: ges 5, 6) 


1) ‘The 50th of sty is assigned 
“Nproblen" or sympton or a provisional diagnosis, ‘the 50th’ 
percentile length of stay or 5 days post admission, which- 

ever occurs first, -is review date assigned. For 

problems or without established criteria sets 
and P.A.S. norms, review te fe 
hours after admission and avery 72 hours thereafter until 
an established diagnosis is indicateds 


* Current diagnosis refers to the diagnostic category which takes prece- © 
dence in L.O.S. assignments on a given study review day. Therefore, if 

a primary diagnosis is entered past the date of review, the admit . 
‘diagnosis is the current diagnosis. 
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| requested for diagnosing and restoring a patient. toa desired state of 


UMI 


2) If a'primary diagnosis is entered in the computer system 

| during the patient's stay, the associated length. of stay 
norm takes precedence over. the: admit diagnosis. | 

3) rf. a noneminor surgical procedure has. been performed and 
is entered, the associated: Length of stay norm takes 
precedence over the. admit and primary diagnoses. 

4) ‘Diécharge diagnoses and procedures and associated iengths 
of stay are documented for comparative analysis of 


\ 
_ predictability of admit versus discharge diagnoses. 


| Levels of Care Data 


A Medical Services Profile is the list of current medical orders 


4 


health. In this study, a current medical order refers to any service 


the healtn care facility is providing, should be prepared to provide if 


a pre-specified condition arises, | or has provided on the day being 
Report Part A is a free text representation of this 
information (for an example see Table 3.1). 

The Technicon MIS Corporation provided a Listing of ald the 


medical orders which can be selected when using MIS. The schemetas 


’ volume of different medical orders far exceeds the 12,000 items on the 


list. This is ae to the fact that most orders cart be modified by 


‘attaching a schedule for performing the services, medication orders: 
specify the medication and the route of pidictieaein. and new orders 


can be generated by existing choices. It seems almost super- 


Fluous to state that one cannot and need not be Capable of uniquely | 
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identifying each possible medical order, at least not for this decision 
process. It is desirable, and in this: study, necessary, to bs to 
identify shedtecash services in ‘ow which provides information for a 
"computerized approximation of level of care decision making. 
|A free text representation of each medical service provides too 
many variables for model development, computer pieand ties of. free text 
| is a difficult problem, and the relative importance of a service for 
| a&cision be not explicit in the free text, Many services are 
- gimilar enough in the bidGenattion they provide for level of care 
assignment that they can be identified by the same name. The reasons 
: for using a classification scheme are panera, I searched for, and 
did not find, an existing anel fication system for medical servites 
which could meet the described need. | 
Thus, a critical objective of this research was the development 
of a classification system for medical services which organizes similar 
services into one class identified by the same code. Due to the com= 
plexity of the process, a detailed description of the dentietion of a us 
classification system and code for medical services io seeekand 
separately in Chapter IV. 
‘ development, Table 4.4 is duplicated as Table 3.2. ‘It lists the pattern 


To permit the reader to follow the discussion of the model 


of predictor variables used to develop a discriminant function. These. 


30 binary variables represent the clusters of medical services that are 


considered useful in describing a patient day in order to determine the 


required level of care. 


For each observation the medical services profile was first 
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TABLE 3,2 Variables and Service Code Cluster Definitions 
variebie | Definition utaiity 
(Intensity 1, Intensity Moderate 
X3 . Nursing (Intensity 
Nursing (Intensity 5) Moderate 
Nursing. (Intensity 7). Moderate 
| Medications (Intensity is IV Intensity 3) High 
| Xe | Medications (Intensity l, IV Intensity 3) os Moderate 
2 Medications (Intensity 3 IM & PO, Intensity 5 High - 
X19 | Medications (Intensity 3 IM & PO, Intensity Gon-r Moderate 
Medications (Intensity 3 IM& PO, Intensity 5)) High 
ie | Medications (Intensity 3 IM& PO, Intensity = Moderate | 
X13. Medications (Intensity High: 
Medications (Intensity 7) Moderate | 
‘Ancillary Diagnostic (Intensities 6) | 
X16 | Ancillary Diagnostic (Intensities l, 3, §) _ Moderate 
Ancillary Diagnostic (Intensity 7) High 
Xie Ancillary Diagnostic (Intensity 7) | Moderate | 
Kis Ancillary Diagnostic (Intensity 9) High: 
‘Ancillary Diagnostic (Intensity 9) "Moderate 
Ancillary Therapeutic (Intensity 1) | High : 
Xoo Ancillary Therapeutic (Intensity 1) : Moderate 
Ancillary Therapeutic (Intensity 3, Intensity 5) High 
Koy | Ancillary Therapeutic (Intensity 3, Intensity 5) Moderate 
Xo5 Ancillary Therapeutic (Intensity 7, Intensity 9) High | 
Xn6 Ancillary Therapeutic (Intensity 7, Intensity 9) Moderate 
X07 Hospital Based service 
Xog Discharge Planning Service 
Physician Rendered Service (Intensities 1, 3) Constant 
X39 Physician Rendered Service (Intensities 5, 7, 9) Constant 


' 
A 
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translated into a coded medical services profile, which was then con- 
verted to the service code cluster pattern--the pattern of predictor 


v 


variables, 


III.3 Application of Discrimination Methods for Level of Care 


‘Allocation 


If the discrimination iadeets and classification evaluation are - 


f 


performed on the samé tai euler! the results are biased. It £6 not 


appropriate to assume that a 4iscriminant model that  aieekeian as an 


acceptable level of accuracy on one sample will be equally successful on 


another sample. To. avoid this criticism, and test the robustness of the 


discriminant model, the stuty population was separated into two sets. 
Every nica observation was chosen for the set on which the model would 


be evaluated. The estimation set consisted of 715 observations aad was 


used to select the best predictor variables and estimate the coefficients 


OF the ane. The evaluation set consisted of 357 observations and was 


used to evaluate a predictive capability of the model. 


As will be discussed in Chapter IV, a list of variables was 


determined by water’ opinion which are “useful in’ making a decision about | 
the ‘level of care required by a patient « on a . specific day. It is to be 


expected that expert opinion will include variables that do not have 


statistical significance for discrimination and thus produce many false 
positives in the variable selection, 
A stepwise linear discriminant procedure was used to obtain a. 


hierarchical selection of the smallest subset of variables from the 


initial pattern. “This method rules out the medical services which are 


ral 
| 
> 


that are retained are combined to give a linear discriminant function 
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9 


non=contributing to the differentiation of the groups. The variables 


used in the decision rule to allocate observations into the two a priori 


determined groups. 


The stepwise linear discriminant procedure was. applied to the 


estimation set. The dependent verianse is the nurse’ Ss judgment on level 


There are 29 predictor’ variables eligible for 


inclusion, variable ae the discharge planning service variable, was 
recorded but was excluded from model development. The URNC, Nurse Coder 


and physicians were asked to disregard this order in decision making 


whenever it occurred, However, it was included en the. service code 


Cluster pattern because it can be used asa predictor: for review 


independent of either.model and should therefore be identifiable for 


separate evaluation, 


The variable which discriminates mest between the two groups (the 


| ‘one with the highest F-value) is the first one chosen to enter the 
equation. variables are chosen ina stepwise 
manner, including at each step the best discriminating variables, while 
! accounting for those already chosen. A variable that has, been “nae 


ten inclusion in the discriminant function can — eliminated if its 


associated F-value (recalculated each time a new variable enters the 


drops below a threshold can occur due to an inter- 


action between new variables chosen and variables already in the discri- 


minant equation. The interaction affects the F-value of previously 


chosen variables. This process for inclusion: (and rejection) s . 


repeated until the F-value for any of the remaining variables.is not 


e 
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statistically significant for inclusion in the discriminant ION. 


The advisability of transforming the linear discriminant function 


into the probability of. allocating individuals into each group has been 
| discussed, The Logistic Discrimination Method is a means of obtaining 


this probability. The variables selected by the linear discriminant 


function were used to fit the Logistic Discriminant Function, The 
Logit, as regression on the predictor variables, is 
computed as the logarithm of the expected proportion of the outcomes, 


The Logistic Discriminant Function was £it by the method: of macimum 


Likelihood. The program used’? calculates the estimated coefficients 


‘of the function, determines the nistlassi Picetion rate, and displays a 


plot of the distribution of the expected Logits for the observations in 


v 


each group the estimation 


Then the en merss Discriminant Model was tested on the ‘evaluation 


set in order to ‘compute a Logit: and allocate | each observation to one of 


the two groups. The observation is allocated with an associated 


probability of membership in the group. The model's allocations will be 


“a 


‘referred to as G, = Model SELECTS FOR REVIEW and Gz = Model SELECTS NOT - 


rs) S 


III.4 Methods for Comparative Evaluation of L.O.C.I. and L.0.S. . 


Selector Models 


Several criteria were evaluated in order to. assess, and thereby 


ad che effectiveness of the Le 0. c. I. and L.0.S. selector models. © 


‘The following criteria were used 
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-a) Correct Allocation Rate 


At this point in time a statistical test of the sige tices 
of an obtained allocation rate does not exist for the 
situation under consideration. Kappa cannot be. used to 
measure the agreement between the discriminant model and the 

‘nurse's judgments because the decisions are not. independent. 
The predictor variables and their associated weights 
representing the relative importance of a variable were 
determined from decisions made by the nurse. Thus, the 

independence assumption does not hold. the correct 
ores rates will be compared as indicators of the predictive 
7 validity of. the models. 

Any selector model's allocations the 


following table: 


"TRUE" ASSIGNMENTS 


G5 
‘PREDICTED 
ALLOCATION 
Gs 


Where:., IP True Positive 


It 


FP = False Positive 
TN = True Negative 
FN = False Negative . 2 


The columns represent the true states as 
determined by expert judgment; the rows 
represent the model's decision. 
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"positive" response by the the model is the 


allocation of an observation to the SELECT FOR REVIEW- group 
(G. ‘or G g) A "negative" response by the expert or the model 
is the allocation of an observation to the SELECT NOT vie 


REVIEW group (Ge « or Ge 3). 


The Correct Allocation Rate is the proportion of True 


- Positives plus True Negatives. 


TP + TN 


Correct Allocation Rate =— 
_ + FP + TN + FN 


Measures of Sensitivity Specificity 


| The Correct Allocation Rate is not a. sufficient measure by 


which to evaluate a selector model. because ut does not provide 


~ knowledge about the types of incorrect allocations. 


Sensi tivity and Specificity are measures that do provide this 


information about the performance of the selector models. 


Selector Sensitivity: the selector sensitivity of a model is 


a measure of how well the procedure under consideration 


selects for review those observatioris which experts determined 


‘should be ‘selected for teview. Therefore, the selector 


sensitivity of a model is a function of the proportion of True 


Positives. 
Selector Sensitivity = = Pr(G.|G.) 
-TP + FN 


The higher the proportion of True Positives, the greater the 


degree of selector sensitivity of a selector mod2l. 


; 
| 
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Selector Specificity: the selector specificity of a model 


is a measure of how well the procedure under consideration 


minimizes the number of observations selected for review 


that experts determined should not be selected for review. 


Therefore, the Selector "specificity of a model is a function 


of the proportion of frue Negatives. 
TN 
Selector Specificity = — > Pr. (Gel¢z) 
TN + FP 


The higher the’ of True the greater 


degree of selector specificity of a ‘selector model. 


Predictive Sensitivity and Specificity: the predictive 


sensitivity and specificity are two other measures of a 


model" S and are relates to selector 


sensitivity selector specificity. The predictive 


sensitivity is a measure os the predictive value of a 


positive response ty model. it. 48 a function of 


the proportion of True Positives and False Hegatives. Thus, - 


| 
Predictive Sensitivity = = Pr (CA . 
| TP + FP 


The predictive specificity is a measure of the predictive 


value of a negative response by the selector model. It isa 


function. of the proportion of True Negatives and False 
Negatives. Thus, 


Predictive Specificity = —— = Pr (c-|G-) 


‘TN + FN 


. > 
‘ 4 

| 

‘ 
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¢) Outcome Analysis of Effectiveness of Selector Model 


An outcome of a judgment without complete information may ‘be 
a Change in assignment when additional information is intro- 
dubed ta ‘the decision making process. Therefore, "outcome" 
is defined is the confident or final decision made by. the 
URNC, 
been developed ‘to predict a decision based 
“Stage 1,2;3 information. If the URNC's changes 


‘often with additional (i.e., Stage 4,5) information, the 


true sensitivity and specificity rates of the selector models 
are affected, Thus, correct allocation rate, sensitivity and 
specificity of each aalactes model will be assessen also for 
the "tonPident op final" level of care assignment--the 


\ 


outcome, 
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types of services available vary between and within each institution. 


af 


CHAPTER IV 
DEVELOPMENT OF _A CLASSIFICATION CODE FOR SERVICES 


IV.O Introduction 


fhe medical services. ordered: cor. a patient are the data used by 


the URNC to decide on the level of care facility which can meet the 


patient's needs: ‘The level of care an institution can provide is a 


function of the services it offers in the delivery of patient care. 


Table 4.1 lists several different level of care facilities and illu- 


strates the services each type of facility is expected to provide. The 


~ Some hospitals have: available for use services which are not. available 


at other institutions, one institution have a cobalt treat- 


é 


facility whereas another may not. Within an 


there is great variation in the types of services available, e.g. 


compared to intravenous medications compared to 


operating room facilities. 


The quantity of different medical services which hearth care 


; facilities offer far exceeds the number oo variables watch the logistic 


discrimination method can handle. Thus, it was imperative to reduce 
the potential number of variables, but in a way that could still permit 
the description of. the level of care a patient requires and the level of 


care an institution provides. 


There are services which share similar characteristics in the 


information they provide for making a level of care decision. Hence, it 
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_ was decided to reduce the quantity of variables by organizing the 


medical services into classes sharing similar characteristics and | 
the classes by a meaningful code. oF the classifica» 


tion ebtorts in ‘the health care field have been in relation to diseases: 


or diagnoses or "surgical socialites A classification systen encoding 


= health care facility environment. in terms ad services it offers. 


‘could not be found. 


IV.1 Introduction to Classification | 


Classification is / recognized as the basis of all scientific 


generalization and is therefore an essential element in statistical 


methodologye Classification 1s aso essential to Gecision 


and. is actually equivalent at a certain juncture in the decision making 


process. It permite the amalgamation of which reduces the 


quantsty to be processed, a factor which. is important as 


2 
quantity of data most. people can use simultaneously is limited. 


Therefore, classification is a reduction process, a means of forming 


sets for deciatans.” The classification of information process is 
iterative util an action decision is a, 

When "classes" of the same type of information are frequently 
used by sain individuals, as is the Case in health care, the process: 


needs to be formalized. A classification system formalizes the process 


to aid in communication. A diagnosis is a name for a set of symptoms 


and Signs} it classifies the set. =: utilized by two Pa the 


diagnostic label Serene the set which is present in the ‘minds of 


the physicians:: By referencing the with a Single label or name, 


| 
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multiple facts determining the label plus Pacts associated with the 
aggregate are transferred one data item. | | | 
_ When developing a classification system, the classification axes 
are by the of the users and information they” 
will need to have about a given class. Therefore, it follows, that the © 


particular axes selected will reflect the interest of the investigator. 


For example, the International: Classification of Diseases’ was adopted 


by the World Health Organ Jon as the staitierd disease Classification 


system and it evolved from initial endeavor by the International 


Statistical Congress in 1853. o individuals, William ssid an’ English 


x . | medical statistician, and Dr.. Marc D* Espine, of Geneva, were. asked to 
a uniform nomenclature and statistical classification 
diseases and causes of death. Berita each. had the same goal in mind, yet 

generated classes based on very different principles. Parr’: S 

) classification was arranged under five groups: Epidemic diseases, 
Constitutional Local diseases arranged according to 
anatomical site, Development: diseases, and diseases that .are the direct ra 

result of violence. D'Espine classified their 

nature (gouty, herpetic, haematic, etc). The classes they formed were 
representative of how each perceived the Clases? cation ‘system to be 


used, plus the pe me and expertise of each one produced a different 


formulation of classes. 


The classes are identified by a nomenclature. A nomenclature is 


a list_.or catalogue of approved terms for describing and recording. 


A. code is a system of rules that enables messages in the source 


. language, i.e., the initial set of information to be transformed into a 
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target language or code language" or 
| Let it be sufficient to state that ‘there are usually four 
reasons to establish a code, and, as already discussed, they were of 


significant tence to the study. They are: 


a). to translate from a. difficult to use source language 
to a language that is easier to use for a particular. 
purpose or purposes, 
b) to decrease amount of space to record information, - 
c) to supplement amount of information available in the 
source language, 
d) to distinguish between alternative ideas or words 
-- that are not easily distinguished in = source 
language.? 


Therefore, not only will a classification system be developed, 


b 


but So will. a codification system to identify the classes, The code 
_. Will provide the ability to use the information inherent in the class’ of 


_ services: to make decisions and assessments about patients and facilities. 


IV.2 Medical Services Classification Analysis 


There are two major approaches to classification analysis: 


discrimination and clustering. 


Discrimination techniques begin with either.a priori Conceptual © 
distinctions or data divided into a priori groups and proceed » 
to develop rules by which to separate data into those a priori 
categories, whereas clustering techniques use a priori 
selection of a measure of similarity, a criterion, and a class 
description to find an inherent empirical structure in data--to_ 
find clusters. Thus, discrimination is the development and 
application of analytical rules, the parameters of which are 
often derived from the data, that allow a researcher to assign 
or sort objects into specific categories. Discrimination uses 
externally supplied labels associated with each member of a 

- set.of objects to aid in establishing rules for sorting things 
into groups. In clustering, which may be either of objects, or _ 
of variables, we seek to find so-called data-derived groups 


a 
‘ 
‘ 
4 


based on internal similarity between the objects; the 
definition of similarity is left to the user, as are. 
procedures by which sorting into similar _— is 

accomplished, © 


both and clustering techniques were used in the develop- 
ment of the’ services classification 

Axes for: ‘Classifying the services were chosen a priori on the 
relative importance attributed to that category « of information. “te 
criteria for determitiing at cone’ give rise to questions which 
are answered by responses which, fall into a combination of the’ 


following categories*: 


‘tatenaity of Service 
Department Responsible for Service Performance 
‘Individual Responsible for Service Performance 
‘Type of 
Two the Department Individual for 
Service Performance were Classified according to the discrimination - 
“approach, Intensity Level and of Service were classified according. 


to the élustering 


a 


‘category - refers to the various axes of which this classification 
system will comprise 

Class refers to the combined four digit code C,.) which 
4 
will define a service | | 

modality — refers to the values a category can assume 
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Category Intensity of Service 


Intensity of service is a function of a combination of 
characteristics of ‘a medical service. A service can be so 
"intense". that it can be provided only on an inpatient 


“basis in an-acute care hospital. If a service is not acute 


hospital based, then it. can range in intensity from 


extremely intense to non-intense. The factors which affect 
intensity are: 
- frequency of service in a 24 hour period; 


- -complexity/skill level required to perform the 
service; 


= risk associated with performance of the service; 


ie: route of administration (for medications and 
diet). 

Medication isceniey level is based on an interaction of the 
specific drug, the route of administration and sien Cnaaiieniy 
of a given dosage level. As a result, medications are | 
considered a nursing service isa Pmiction 
and/or a high risk medication. ‘Each medication was scaled 
ona risk factor. If the medication: itcers had a higher risk 


° 
than the route, the medication took precedence in intensity 


assignment. 
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‘The following codes identify the intensity category 


modalities: 


(0) Hospital based service 

1) Extremely intense service 

(3) Very intense service i 

5) ‘Moderately intense service | 

7) Minimally intense service 
(9) Non-intense or trivial intensity service 


‘Category 2: D epar tment Responsible for Service Performance 


This category should be evident from “the The 


Clinical Pathology anes all the laboratory par pathology 


work. ‘The Radiology Department does all the x-ray 


diagnostic and therapeutic work. Other departments, 


similarly, perform designated sets of medical services. 


‘Department responsible for service-performance is already 


encoded in MIS. When an order is written by a physician, 


the system notifies the appropriate department(s) that the 


service has been requested, 


The following codes identify the modalities for this’ - 


category: 


) 
) 
) 
) 
) 
) 
) 
) 
) 


Clinical 

Radiology (includes Radiation Therapy ona 
Nuclear Medicine 

Nursing 
Rehabilitative Therapies (includes Physical and 
Occupational therapy) 


and Respiratory Therapy) 


Electrocardiogram, Electroencephelogram or 
Electromyelogram Departments 

Discharge Planning 

Other 


Respiratory Medicine (includes Pulmonary Medicine 


Surgery or any private physician rendered service 
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For example, in the Nursing Department there are some- 


services that must be rendered by an R.N., whereas other 


Category 3: Individual Responsible for Service Performance 


This category specifies the individual within the Department oe / 


who assumes the responsibility. for the requested service. 


) 


services can be under the R.N.'s supervision. For Clinical | 
Pathology, one assumes that a’ technologist 

the test and reports the results unless 
is required to: This category is clearly defined by the 
rules of each department and/or the laws of the state. 


The following codes identify the modalities for this 


Category: | 


1) Registered Nurse (R.N. ) 
2) Under R.N. Supervision (includes 
Nurse's Aides, Orderlies, Central Supply) 
(4) . Therapists, Technologists, Technician - 
specially trained (and usually licensed) 
individual within an ancillary Tepersnens.. 
(5) Dietician 
(9)’ no special skills can be members of 
the patient's family or the-patient 


Category 4: Type of Service 


The type of service is the dominant purpose of the order. 


| 
( 

| 

| | 
| t | 
| 
| 
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The following codes identify the modalities of this 


category: 


(0) routine service, routine measurement, support 
service 
ey observation/interpretation including moni toring 
(2) diagnostic 
therapeutic 
(4) psychological/psychiatric 
Al Traction and patient status orders © 
(6) precaution and isolation 
(7) medication (includes infusions and transfusions) | 
(8) procedure - surgical and non-surgical 
(9) other | 


Table 4.2 summarizes the definitions of the classification 


system. The combination the four digit service code which 


labels each medical order. 


It is important to note that the “intensity of service" digit is 


based on an ordinal scale whereas the digits used 


Cay are ‘based ona nominal scale. the services with an intensity 


digit of 0 or 1 have a higher intensity associated with them than those 


with an ‘intensity of 9. There is no indication how ‘mach tore iabaniee 


one level of intensity is than another or even an absolute value of the 


intensity for a given level. It should be recognized that ordinal 


scaled values cannot be manipulated mathenatically but do convey 
| information: on the relative relationships between objects. Nominal 


scales are used to label or identify only. There is no relationship. 


between the relative position of the digit used and the attributes of 


the objects being labeled, thus nominal scales also cannot be 


manipulated mathematically. 
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TABLE 4,2 Classification end Codification System for Medical Services 


A four digit code which classifies services provided by health care 
facilities and health care practitioners, Each digit provides infor- — 


mation about services which are in a class defined ne C,;C5C3C,. 


CATEGORY (Cy 


“CATEGORY 2 (C5): 


NOW PWN 


CATEGORY 3 (C3): 


CATEGORY 4 (C4): 


0 


WN 


| INTENSITY LEVEL 


‘Hospital Based Service 
Extremely Intense Service 


Very Intense Service 
Moderately Intense Service 


Minimally Intense Service 
Non=Intense or Trivially Intense Service 


DEPARTMENT RESPONSIBLE FOR SERVICE PERFORMANCE _ 


Clinical Pathology 
Radiology 


Nursing 


Rehabilitative fherepies 


Respiratory Therapy 


Surgery or any Private ‘Rencerec Service 
Electrocardiogram, Electrosncephelogrem, 


Electromyelogram 


Discharge 


Other 


INDIVIDUAL RESPONSIBLE, FOR SERVICE PERFORMANCE 


Registered ) 


Under R.N, Supervision 


Therapist, Technologist, Technician 
Dietitian 


No Special Skills Patient, Family 


TYPE OF SERVICE 


Routine Service, Routine Measurement, Support Services. 
Observation, Interpretation, Includes Monitoring 
Diagnostic. 
Therapeutic 

Psychological/ Psychiatric 

Traction, Patient Status Orders 

Precaution and Isolation 

Medications, Infusions, Transfusions. 


_ Procedures ~ surgical and non-surgical 


Other 


‘ 
> 

= ‘ 
= 
= 
= 

= 
= 

. 
| 
> 
4 6) 


the four categories. Since it is the "physician who requests the 


Classified into modalities that are distant from each other. 


IV.3 Classifying the Medical Services 


7 Once ‘the critical modalities were defined, every hospital service 


4 


that can be ordered by a oaysician was classified —- to each of 


services to be provided to a patient, it follows that the source - 
warabatene’ be all possible physician orders. The Listing of all 


physician orders currently used was supplied by the Technicon MIS 


Health care professionals (two physicians and two nurses). 


. trained in levels-of-care concepts Classified each of the 12,000 medical 


services into each of the above —. Each person worked 


independently. After all the services were classified, the iit met to 


compare Classifications. — Services with total classification agreement 


were not’ discussed one an individual raised feelings of doubt. For 


those services with classification disagreement on aut open. 


discussion almost ses led to CONSENSUS » 


An expert in each "department responsible for service 
performance" classified the medical services in nis/ner domain of 
responsibility. There was a high level of agreement between the 


departmental expert and the above referred to health care professionals. Pee: 


_ Any disagreement was in the intensity of service category. When the 
modalities assigned to were Close, the departmental expert's category 


took precedence. There were few occurrences of a service being ; . 


The health care professionals were then presented with a list of | 


4 digit codes and the services that had’ been classified into those 


81 
| 

= 


UMI 


82 


| Classes. They were asked to confirm that the services ina class" 


belonged together, that is, ‘that ‘they ‘provides the same kind of 


information for leven of care assessment. About 10% of the services 
were reclassified. This was repeated there was full 


consensus. The Seem — then’ reviewed these final lists of 


classified found.less than 1% 


~ 


Appendix A isa dictionary ofa sample of the medical services and 


' associated codes. It is in departmental order, 


The system was designed to permit ‘expansion when the codes within 


a category are no longer capable of adequately defining a service, This 


means.that if a new medical service is developed and does not Craarsy i 


fit into an existing category with its associated code, a new one will: 


be added. Ina field like medicine, where the knowledge explosion has 


‘been very rapid, and is still expanding quickly, the system needs to be 


easily expandable to meet new information requirements. If it is not, 


it wottld very quickly lose its utility and significance. 


% 


IV.4 Codification of Medical Services Profiles © 


A turse (Nurse Coder) was trained to apply the codes to the 
medical services medical services for each 
observation were translated into the associated 4 digit code thereby 
generating a coded medical services profile, 


» 


On careful examination of the medical services profile (Report I 


‘Part A) an interesting fact was observed. The date each order is 


written is included on the report. The URNC uses many of the orders 


= 
| 
‘ 
' 


with a decrease in importance as the age of the order increases. Some 
orders are never discontinued al though the may be close to 
zero “that. the Patient the service. This is especially 
jetiembie fos PRN or conditional orders, The age and PRN/non-PRN 
affected the relative utility of a order to the decision making | 
Process, and therefore were also documented in the coded medical 
services profile. ‘Table 4. 3 illustrates the Conversion from a medical 
services profile to a coded medical services profile, 
once all the observations were coded, we fowumd that 80 different 
service codes were actually’ used. This quantity quickly multiplies when 
the age of the order wid PRN/non-PRN factors are. combined with the 
service code. In addition, an individual could have multiple occurrences 
of’ the same service code with the sane and/or 4ifverent age and “PRN 
factors. The number of possible variables was still. too great to perform 
| the analyses discussed in. Chapter tI. Could the service codes be combined 
a fashior, reducing the number of variables but 


the information needed for ¢ level of care ‘selector model? 


-IV.5 Conversion from Service Classes to Service Clusters - 


To deterinine if the service si nikon could be combined and stilt 
provide dhe information necessary to make a level of care decision, the 
author and one of ‘the physicians, involved in the code development 


ee | studied the list of 80 service codes actually ‘used and the proportion of 


use in both the SELECT and SELECT NOT FOR REVIEW groups. Seven service 


Clusters emerged as dominant factors which were labeled: 


one 
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Nursing (medications excluded) 

Ancillary ‘Diagnostic 

Therapeutic 

Based 

Physician Rendered Service 

Discharge Planning 

These service clusters reflect the types of criteria which are © 
used in making levels-of-care. assessment, as illustrated 

care decision flowsheet (Figure 3.3). This indicates that the service 


codes are useful in describing patient needs in terms of level of care. 


The service clusters were then presentea to the. Nurse Coder for 
“subjective evaluation of the usefulness or utility they had in her 
decision-making. She that she could split the clusters if 
important information was lost in the combination or that she could. 
combine category modalities further. | 
| The following service clusters were split due estes intensity 
| 


(medications excluded) 


Nursing 


| Nursing-Intensity 1 Nursing-Intensity 5 Nursing-Intensity 7 
| Intensity 3 
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Medications 


Medications Medications 
-Intensity 1 -Intensity 3 -Intensity 7 
-Intensity 3 | (IM & PO) 
(IV route) _ -Intensity 5 
PRN NON-PRN 


[Ancillary Diagnostic] 


Ancillary lagnostic Ancillary Diagnostic Ancillary Diagnostic 


: “wIntensity 1 -Intensity 7 «Intensity 9 . 


Therapeutics} 


Ancillary Therapeutics Ancillary Therapeutics Ancillary Therapeutics 
-Intensity 1  :eIntensity 3 -Intensity 7 
-Intensity 5 . Intensity 9 


Physician rendered service Physician rendered service 
-Intensity 3. -Intensity 7 


-Intensity 


| (Nursing Intensity 9 and Medications Intensity 9 were eliminated because 


UMI . 


they had little or no utility in decision making.) oo | | | | 


sician Rendered Service 
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~The 80 service codes were now reduced to, seventeen useful dicho- 


tomous variables, the 15 above + Hospital Based Service + Discharge 


Planning Service. It had already been determined that the age and PRY/ , 


non—-P RN factor affected the utility a a service Code to the level of - 


care decision. The quantity of service codes in a service cluster also 


affected the utility. ‘Physician rendered services have constant 


uti li 


. For each new eabiibte. its relative utility for the level of care 3 
obtained as a function of age and PRN/non-PRN, A separate 
utility curve was obtained as a function of the quantity of swevice 

codes that. contributed. to the baascasanss of that variable. For the age- 
factor utility curves, age equal to zero was er a utility of oo... ” 


A zero or near zero “utility level was assigned to some paras further out 


in time. A curve connecting the two points was drawn after Some inter- 


mediate paneer were directly obtained. The slope of the curve and the 
Point were dependent on the specific service Cluster ‘ana. the PRY 
non-PRN Factor. 


“The same procedure was | repeated for the quantity factor, only 


this time, assigning to quantity zero the zero utility ectilt and a 


| utility of 100 to a quantity greater than zero but’ dependent on the 


specific service cluster and the PRN/non-PRN factor. 
Figures Ja-13a in Appendix B represent the utility curves for a 


service chenver: 9 a function of age, PRN/non-PRN. It is the utility of 


“service cluster s fax age a, 1q-13q in Appendix B represent 


the utility curves for a service cluster: as a fection of scr It 


is the utility of service cluster. s for quantity q, Ueq°. 


The final objective was to determine a utility level for a 
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Service cluster as a function of both age and PRN/non-PRN AND quantity, 


Fe a Could one assess the utility of a multi-attributed consequence 
by assessing the utility of each attribute and then forming a weighted 
average?  A.critical aspect is the independence of the attributes. In 


this problem, the attributes are not independent. 


Neither U nor U ce sq accurately represented 


Sa sq Saq sa 


or even approximated the plane defined by the combines utility of both 


attributes for each service ctustens. 


Since many of the utility curves resembled step functions, it was 


decided that there ‘wuie not be a significant iéea of information if 


was into 3 regions of utility: high, moderate, low. 


For ae Sieve we =e scores 2 80 were coded as high, a utility 


| score > 39 and < 80 was coded | as moderate and a utility score < 40 was 


coded Rules were determined for combining the. regions on ‘the 


9 


two attributes. The fact that age-PRN/non-PRN has greater wratiey this 


quantity is reflected in the rules. The following matrix represents the 


possible combinations: 


| QUANTITY FACTOR UTILity 


Q-HIGH . Q-MODERATE Q-LOW 
8 
A-MODE RATE Moderate Moderate Moderate 
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Each service cluster was represented by two dichotomous 


variables, 


sy = 1 & Xqy.= 0 indicates High utility of service cluster S 


| 


oH = 0 & Xoy = 1 indicates Moderate utility of service cluster S$ 


sy = 0 & Xgy = 0 indicates Low to No utility of service cluster S 
where subscript SH refers to high utility for service cluster S_ 
subscript SM refers to moderate utility for service cluster & 


f 


The rules were applied to the service code profile of each 


observation and ioovertes it to a service code cluster pattern of ~~ 


dichotomous X. variables which are listed in Table 4. 4. ‘the conversion 
of a medical services profile to a’ service, code profile has been 
illustrated... Table 4.5 illustrates the same service code profile. 


conten to a service code cluster pattern. 


IV.6 Summary of Classification of Medical Services 


A rationale and approach for the development of a Classification i 


System for medical services was Discrimination and 


clustering techniques were utilized to derive rae to aid in ‘the 
classification process. Medical service codes identify classes of 
medical services. | 

The service codes are qualitative representations of a patient's 
current orders. Because the number of possible variables was ee 


great for model development, a rule was developed for combining service 


codes (while adjusting for age of order and PRN/non-PRN factors) into 


pees service code clusters which could still be used to describe an 
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TABLE 4,4 Variables and Service Code Cluster Definitions 


— X39 Physician Rendered Service (Intensities 5, 7, 9) 


Variable | Definition 
Nursing (Intensity Intensity 3) High 
Xo Nursing (Intensity 1, Intensity 3) ‘Moderate 
X3 Nursing (Intensity 5) _ High 1 
Xy Nursing (Intensity 5) Moderate 
Ks Nursing (Intensity 7) High 
Nursing (Intensity 7) Moderate 

X7 Meddcations (Intensity 1, IV Intensity 3) 

Medications (Intensity 1, IV Intensity 3) Moderate 

Xe Medications (Intensity 3 IM & PO, Intensity 5)) High 
Medications (Intensity 3 IM & PO, 
X11“ Medications (Intensity 3 IM & PO, Intensity tren High 
X12 Medications (Intensity 3 IM & PO, Intensity 5) Moderate | 

on ee Medications (Intensity 7) ‘Moderate 
X15 Ancillary Diagnostic (Intensities ‘High 
Xig~ Ancillary Diagnostic (Intensities 1; 3, 5) Moderate 
X17 | Ancillary Diagnostic (Intensity 7) High 

Xa Ancillary Diagnostic (Intensity 7) Moderate 
lary Diagnostic (Intensity 9) ‘High | 
X90 Ancillary Diagnostic ty Moderate 
X21 Ancillary Therapeutic (Intensity 1) High 

Ancillary Therapeutic (Intensity 1) ‘Moderate 
X23 Ancillary Therapeutic (Intensity 3, Intensity High 

“Xo & Ancillary Therapeutic. (Intensity 3, Intensity 5) Moderate 
X25 2 Ancillary Therapeutic ‘(Intensity 7, Intensity 9) High 
Ancillary Therapeutic (Intensity 7, Intensity 9) Moderate 
X27 Hospital Based Service | 

Xoq Physician Rendered Service (Intensities 1, 3) Constant 


Constant 


a 
‘ 
‘ ‘ 
w 
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TABLE 4.5 Conversion From the Service Code Profile (exemple) ‘to a Service Code Cluster Pettern 


SERVICE 


SERVICE CODE 
CODE CLUSTER VARIABLE VARIABLE 
PROFILE PATTERN . NUMBER NAME UTILITY 
Nursing (Intensity 1, Intensity 3) High 
1317 0 Xo | ‘Nursing (Intensity 1, Intensity 3) ‘Moderete 
X3 Nursing (Intensity 5) High 
Xy Nursing (Intensity 5) Moderete 
0 Xs Nursing (Intensity 7) High 
sw 1 X7 | Medications (Intensity 1, IV Intensity 3) im 
Xp Medications (Intensity 1); IV Intensity 3) Moderate 
0 Medications (Intensity 3 IM & PO, Intensity 5) * 
0 X10 Medications (Intensity 3 IM & PO, Intensity 5) - Moderate 
ee Medications (Intensity 3 IM PO, Intensity 5) High 
0 X12 Medications (Intensity & PO, Intensity ‘Moderate 
X13 Medicetions (Intensity 7) High 
1 Medications (Intensity 7) Moderate 
| Kis Ancillary Diagnostic (Intensities 1, 3, 5) “High 
a 0 Xi Ancillary Diagnostic (Intensities 1, 3, 5) Moderate. 
= X17 Ancillary Diagnostic (Intensity 7) » High 
0 Ancillery Diagnostic (Intensity 7) Moderate 
1 X39 Ancillary Diagnostic (Intensity 9) “High 
X20 Ancillary Diagnostic (Intensity 9) Moderate . 
9 ‘Ancillary Therapeutic (Intensity 1) High 
0 X20 Ancillary Therapeutic (Intensity 1) Moderate 
X03 ~ ancillary Therapeutic (Intensity 3, Intensity 5) “High 
0 Xou Ancillary Therapeutic (Intensity 3, Intensity 5) Moderate | 
3 0 “Xa5 Ancillery Therapeutic (Intensity 7, Intensity 9) High 
oo 0 X26 Ancillery Therapeutic (Intensity 7, Intensity 9) Moderate 
| Hospitel Besed Service | 
0 Discharge Planning Service 
0 Physician Rendered Service (Intensities 1,3) Constant 
X30 


Physician Rendered Service (Intensities 5,7,9) 


Constent 


. 
u 
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92 
appropriately utilized acute hospital patient day.” The “ability to 
describe" was the criterion for rule acceptance. 

A binary variable for each service code cluster was thus created 
to represent the presence or absence of each combination. By applying 


the rule to a coded medical services profile, a service code cluster 


pattern can be derived. The service code clustgr pattern is the 
complete set of possible clusters of service codes used to describe’ - 


each patient day. Figure 4.1] is the schematic representation of the 


Classification and clustering process. Each service code cluster is a 


7 potential predictor variable in the level of care decision model. 
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ORDERS CODED MEDICAL SERVICES 


Date-Current Order Age-PRN/non=PRN 
|Date-Current Order | Age-PRN/non-PRN 
Date-Current Order Age-PRN/non=PRN 


Date-Current Order | 
\Date-Current Order Age-PRN/non-PRN 
|\Date-Current Order | Age-PRN/non=PRN 


Date-Current Order 


Lililt 


C;CoC3C, 


Age-PRN/non=PRN- 


CyCoC3C, 
| Age=PRN/non=-PRN 
Oate~Current Order | Age-PRN/non=PRN 


3Cy, 
3Cy, 


CiCoC3C, 
C,CoC3Cy, 


~~ 


SCHEMATIC REPRESENTATION OF CONVERSION FROM 
FIGURE 4,1. CURRENT ORUJERS TO SERVICE CODE CLUSTER PATTERN 


SERVICE CODE 


USTER PATTERN 
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X20 
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X25 
X26 | 
X97 
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observations, both by nursing unit and estimation/evaluation sets. 
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CHAPTER V RESULTS 


V.O Introduction 


A description of the — sample is presented Mrst, including 
the findings on the information sources actually used for decision | 
making. Reliability of decision makina between bees and between 


physicians, as well as predictive WaLAGS Sy of ‘the nurses' judgments is 


. established, Pollowea by a: discussion of the development of. a -discri- 


minant model based on data that is reliable and valid for the 


decision process under study. Then, the results of comparing the . 


information provided by the discriminant model - L.0.C.I. - and the 


L.0.S. selector model are presented, thereby providing an evaluation of 


the models. 


V.1 Description of the Sample — 


= 


Although the sampling period started in Jue, an observed lack 


of consistency in the nurse's decision making necessitated rejecting the 


data fron the first two The base used for model develop- 
ment and evaluation consisted of 1072 inline representing 488 


different patients. Table 561 summarizes the breakdown of the total 


Table 5.2 summarizes the number and the proportion of observations in 


groups G, = SELECT FOR REVIEW, and Gz = SELECT NOT FOR REVIEW for each 
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‘to the SELECT FOR REVIEW group, G 


97 
The mean length of stay of the patients in the study is 10.5 


days; the median (or 50th percentile) is 7 days; and the mode (or most 


frequent) is 5 days. Figure 5.1 illustrates the distribution of lengths 
of stay (the sixteen lengths of stay 2 > 34 days were shades as 34 
days for display purposes only). When — — computer-based 


“information “oF decision making, 55% of the observations were assigned 


s* The URNC obtained further informa- 


: tion to make a final decision for 353 out of these 590 observations in 


G.. That is, she consulted the chart and/or other health personnel 33% 


of the time. oe ‘non-computer-based information was added, the con—— 
fident level of Care assignment. became acute care hospital facility for 
248 out of the 353 observations. Hence, 77% of the URNC decisions were . 


based on information available in the computer-based medical record. 


V.2 Evaluation of Inter-Reviewer Agreement on Level of Care 


Assignments 


V.2.1 -Inter-Nurse-Reviewer Reliability 


The importance of establishing has been discussed in 


Chapter Ill. Two nurse reviewers (URNC and NC) applied ‘levels of care 


criteria to the sampled population and arrived: at independent 


on level of care assignments for each observation. If the degree of. 


agreement between the nurse reviewers did not differysignificantly from 


' zero,.as measured by Kappa (K), then the nurses’ decisions are not. 


reliable and a meaningful model could not be developed from the data. 


The notation which will be used to denote group assignments by 
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the nurse reviewers is as follows: 


URNC: G 


= SELECT FOR REVIEW 
8,0 
Ge SELECT NOT FOR REVIEW 
NC: G, = SELECT FOR REVIEV 
’ 
Ge = SELECT NOT FOR -REVIEW 
| 


The following hypothesis was tested: 
H Inter Nurse Reviewer Reliability (Agreement) Equals Zero, K,, =0 
‘The null hypothesis was tested three times: once each for the 
entire study sample for the and for the 
evaluation set (Kw, ee Tables 5.3, ae “6 present the findings. for the 


reliability assessment of the inter-nurse-revi ewer “assignments for the 


entire study sample, the estimation set and the eaeties set, 


respectively. 


The Kappa’ Ss obtained for the study sample, estimation set and. 
evaluation set are each significantly different from zero at the .01 


evel, 1. probability of obtaining this. value of Kappa chance 


alone. is less than 1%. The null hypothesis therefore can be rejected, 


and we can state that there is inter-nurse-reviewer agreement. The 
nurses’ assignments to the two groups are mutually consistent, i.e., the 


judgments are reliable. Kyu and Were evaluated to ascertain 


that the separation into the two sets produced samples that were equi- © 
valent in reliability of judgments. Testing the difference between the 


two Kappas we obtain a z = 309 which is not significant at the 205 level, 


and the possibility that the two Kappas come from the same population © 


Cannot be réjected. 
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The reliability of nurse reviewer decision making is not very 


surprising. Both have training and experience in levels of ‘care crite- 


ria and the rules for applying the criteria. Previous studies 


assessing ‘the reliability of utilization review coordinators decisions 


arrived at the same conclusions. Having established reliability in the 


nurse reviewers' judgments, the predictive validity of the decisions | 


must be eStablished. If the nurses agree with each other, but do not 
provide a valid screen for the utilization review physicians, they are 
not fulfilling their true function. 


V.2.2 Predictive Validity of Nurse Reviewers’ Judgments 


An instrument used for prediction is evaluated by an established 


criterion of the behavior one is trying to predict. An objective 


"correct" answer for level of care assignment does not exist. Given 


that the level of care concept developed from progressive patient care 


principles which were then translated into legislative dictate, the only 


"correct" answer is the one agreed upon by PSRO physicians. Therefore, 


physicians' (with Utilization Review experience) judgments have been used 
as the criterion Sy which to evaluate. the nurse reviewers' judgments. 


The stability of the criterion was established by the use of Kappa to 


measure the reliability of physicians' judgments. 


V.2.2.a Inter-Physician Reviewer Reliability 


The following notation will be used to denote the judgments of 


‘tthe physician reviewers: . 
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Physician-1: = SELECT FOR REVIEW 


S,P1 | 
G- ., = SELECT NOT FOR REVIEW 
§,P1 
Physician-2: G, - = SELECT FOR REVIEW 
? 7 
85 pp SELECT NOT FOR REVIEW 


The null hypothesis ~—_* is tested to establish reliability in physician 


level of care decision making is as follows: 


Ho? There Is No Physician Consensus on Level of Care. Assignments,. 


Xx 


P1P2 
The physician level of care judgments and the measure of agree- 

ment are summarized in Table 5.5. 
' The'physicians are reliable in their level of care assignments. 


Kp1po is significantly different from zero at the .01 level and the null 


is The degree of between the physicians 


is not as high as between the nurses, A substantial 


proportion of the disagreements isa consequence of marginal discre-_ 
investigated by reviewing the comments 
the physicians had been asked to record as they were making their 
decisions. Physician-2 frequently questioned the appropriateness of 
medical services whit co then affected his level of care ééci sion. The 
physicians had been asked to make a utilization review Paver eee that of 
appropriateness of location based on provided. ‘The 
validity of using this ‘assumption: had been established by a previous 
study which tested the hypothesis that the services ordered and used 
in determining the level of care were not unreasonable or inappropriate 


to the patients’ medical care needs. Therefore, Physician-2 was making 


‘an’ unnecessary and biased decision in view of the objective. This 
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assigning 12% more rial the observations to 'G 


V.2.2.b Nurse-Physician Agreement 
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‘addi tional factor in arriving ata judgment resulted in Physician-2 


than assigned by 


Physician-1. ‘Due to this consistent bias, and the fact that reliability 


of physicians’ judgments had been established anyway, Physician-1's 


judgments were used as the criterion for a "correct" level of care 


assignment. 


The following hypothesis tests the predictive validity of the 
URNC's judgments: 
Ho? There Is No Agreement on Level of Care Assignments Between atursy 


and Physician Reviewers, Kyp = . 


. Tf the null hypothesis is rejected, it can be said that the nurse 
reviewers predict well the decisions that physician. reviewers would 
arrive at. Therefore, they can act as physician representatives and 


screen patients for physician review. ‘The degree of predictive validity. " 


isa function of the *value of the coefficient of agreenent which 


: measures, the correlation of categorical assignments between the instru- 


ment and the criterion. Table 5.6 summarizes the results. of nurse- 
physician level of care judgments. (Since the nurse reviewers are con- 
sidered paliadie makers, one could choose the assigriments oF 


either one for (1) assessing the nurse-physician-reviewer agreement level 


and (2) developing the discriminant function, ) 


The Kup is significant at the £01 level and the null hypothesis. 


is rejected. It can be observed that the’ disagreements that ‘occur are 


usually in the case of the nurse reviewer assigning observations to the 
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SELECT FOR REVIEW nia that the physician judges as appropriate, beOos 
False In view of the intent of the selector system, an 
acceptable proportion of False Positives is not as serious as False 
Negatives. The predictive validity, of the nurse reviewers' judgments | 
is highly: satisfactory; she isa good predictor of those Cases the URMD 
would want selected for further review and those, which can be considered ; 


appropriate. 


© Summary of Reliability and Validity Results 


| Nurse reviewer reliability and predictive validity have been 
established. Having: done so, we can assume that observations which will | 
be used to determine function been assigned ins 
consistent fashion and satisfy the intent of the assignment procedure, 
This assumption. is important since the vertu 4 level of the discri- 


minant eunction is related to the wrap deat with which the observations 


were originally assigned. Given that an absolute "correct" answer does 
not exist, the coefficients of agreement are high enough to permit con- 


fidence in the data base. . 


V.3 .A Discriminant Model for Level of Care Allocation | 


A pattern of 30 binary variables naa Deen established to represent 
the various types of medical services used in making level of care 


A linear discriminant program” was 


used to determine which of the binary variables are most useful in 


discriminating SELECT FOR ‘REVIEW from SELECT NOT POR REVIEW observations, 
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Nine variables were chosen and are listed in Table 5.7. The results at 
each step of the stepwise linear discriminant analysis program are 
included in Appendix C. 


The F-values for inclusion of variables X and Xn6 are low. | 


20 

‘te Classification matrix of. the: linear discriminant analysis program 

after Step 7 is almost identical to the classifiéation matrix after 

Step 9. Given that the goal is to use the smallest subset of variables 
which a desired level ob ability, the incremental 
@iscriminating value of these two variables was questioned. Therefore, 
a logistic discriminant Punction was fit using the first seven variables 
selected by the stepwise linear discriminant function, as well as for 
all nine recommended variables.’ The accuracy of allocation for the 
seven-variable and nirewariable logistic functions vere ‘equivalent. 
Since there is no improvement in discrimination. powers with variables 

and included in the model, and an to determine the 
smallest Subset of variables to discriminaté satisfactorily, the seven 
variable-model was chosen for evaluation. "Ve ee D for results| 


of fitting logistic function using seven variables.) 


. The logistic discriminant function was developed from the esti- 
mation set and tested on the evaluation set. The coefficients were 


Calculated by the maximum likelihood method.” A Logit of p is © 


Calculated for each observation, and is the Level of Care Index. The 


value of the L.0O.C.I. is'used’ to allocate an observation to one of the 
two groups, or Gee The L.0.C.I. is Calculated by the following 


equation: 
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_ If another patient had additional service codes denoted by variable X 
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ln (-): -.942 + + «601X3 + + 573X1] 


where a value for L.0.C.I. less than zero suggests possible mis— 


utilization of an acute care hospital facility, and the observation is aie 


then allocated to G..(see Appendix E for individual results, ) 


Inspection of the coefficients indicates that with the exception of 


variable Xoq (the hospital based service variable) at least two and 


sometimes three variables must be present to allocate an observation to 


Gee The coefficients represent the weight, or amount of increase in 


| Ly0.C.I. caused by the presence of the corresponding variable. In 3 


addition, the estimate of the posterior probability P can be calculated. 


Thus,’ for example; a patient receiving variable X, only, would have a 


= and would be allocated to Gz with a Pr(G,,| X,) = .52*. 


S 


the L.O.C.I. would = + .813, and the ‘patient would be allocated to Ge 7 


witha Pr(Gz|X,X.) = .84*, The additional variable, increased the 


= 


probability that the patient required an acute hospital from .48 to .84. 
The results of fitting the. logistic discriminant, function using the 


estimation set are summarized in Table 5.8. 


* The logistic discriminant program used in this research sets 


Y= +1n(—) which only affects the range of p values. 
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The misclassification rate for ‘the estimation set is 22%. 

Norusis _ Jacquez state that a probability of correct classification 

in the range of 80% j is often acceptable. to many investigators when ‘the 

. groups are not clearly defined. Ina population where the groups are 
clearly differentiable, a higher correct classification rate would be 
sought. Level of care assignments are a oP various cut-off 
points on a continuum of medical service needs. Observations at either oe 
end of the continuum are easily differentiated, however many | 
observations PIaSTURTS around the middle of the continuum and therefore 
are more difficult to decide upon. These are also the observations 

upon which the reviewers tended to disagree, | 

| ‘Since the objective was to see if ‘ saci capturing the 
reviewers judgments could be aie: and if one could, “whether it 


would predict more accurately than L.O. Se which | observations to review, 


model development was suspended in to bettors the 


? evaluation. If the L.O. C, I. was considered | a better predictor than the, 


L. O- S. selector model, then-one can only assume that further development 


work would only enhance its discriminating abilities. 


V.4 Comparative Evaluation of L.0.C.I. and L.O.S. 


The effectiveness of each of the models is a function of correct 
allocation rate, sensitivity, specificity, and outcome analysis. The 


degree to which the model (hs 0.C.I. ) approximates reviewers’ judg~ 


ments is described by. the correct allocation rate. The results of 


applying L.O.C.I. to the entire evaluation set are presented first. 


However, having documented only the length of stay guidelines for the 
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50th and 75th nn and not those for the 90th, a bias in favor of 


rd Le 0.C,. I. ‘would be inerdauced when comparing the two selector models. 


This is due to the fact that observations eligible for assessment by 
L.0.C. I. are no longer for L.0. if the actual Length of ‘Stay 


is beyond the 75th percentile forthe current diagnosis or surgery (when | 


assigning the length of stay diagnosis current is a@efined as the 


diagnosis entered in the computer system at the time of review). 
Therefore, all subsequent evaluation results. are Per the subset of the 
evaluation set whose actual length of stay was less than or poet to the 


75th percentile of their current diagnosis. The evaluations were con-_ 


' ducted for the current diagnosis and the discharge diagnosis separately. 


This was done because of the high percentage of Change between admit and 


discharge diagnoses as is presented in Table 5.9. . The accuracy of the 


- information used in either model affects the accuracy of ‘the 


| predictions. Thus, if by using song. of stay guidelines for discharge 


diagnoses Detter tar found, then greater emphasis might 
be Placed | on entering the primary diagnoses as early as possible or. 
changing ‘the to reflect the admitting diagnosis. As a result, 
7% and 75% of the observations in the evaluation set were included for 


evaluation when the cut-off was ‘the 75th percentile by the current and 


V.4.1 Evaluation of Allocations Pre and Fost the 75th 


Percentile 


The comparative evaluation of L.0.C.1. and L.0.S. will be per- 


formed for the subsets of the evaluation set which include only those 
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observations whose length of — is re to or less than the 75th 


percentile of the current and discharge diagnoses. It was desirable to 


confirtn whether bi 0. C.I. _ equivalent discriminating ability before 


and the 75th for either diagnosis. 


The chi-square (2) test” was used to compare allocation 


matrices pre and post the 75th percentile, - the 2 test isa method oF 


comparing the proportions of two samples. It permits the null 


hypothesis that there is no relationship between the model's allocations } 


and the pre and post 75th percentile subsets, that is, that the accuracy | 


of allocations is independent of the actual length of stay relative to 


the length of stay norms. 


: Tables 5.10 and 5.11 are the contingency tables representing. the 


) 


allocation matrices for the pre and font he percentile Subsets for 


current and discharge diagnoses’ The obtained are highly 


significant ant the null hypotheses are rejected. The allocations are 


neither using current nor If each group is 


examined separately, that is. looking at G. separately — Ge, the 
x2 = 39 (1.32)*, p> .05 Pound for G., does not permit the rejection of 


the null hypothesis,, whereas x? = 12.49 (13.78)*, p< .01, for Ge does, 


‘This can be interpreted as indicating that L.O.C.I. selects patients for 


0 


review who should be reviewed equally well before and after the 75th 


percentile (for either diagnosis), -however, for patients who need not be 


-%* Results in parentheses are for subset defined by discharge diagnosis. 
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selected, there is a difference in L.0.C.I."s discriminating powers 


between the subsets. The effects of these differences will be discussed 
in certs of the’ effects on the measures of “performance. 
os : Tables. 5.12 and 5.13 present the allocation matrices for the 
set, for the above aefinea subsets, as well as summarizing 
the measures of performance roe each. 
_ The Correct Allocation Rate decreases several percentage p< points 
: in the post 75th percentile subset, The selector sensitivity increases 
slightly, and the sensitivity remains This latter 
result is important because it indicates that the predictive: value of a 
Le 0. selected patient is and post the 5th percentile 
cubeeths A patient selected for review by L.O.C. I. has approximately 
an 80% probability of requiring ‘further review in either subset. 
However, ‘the specificity levels of I L.O.C. 1. are affected by the patient's 
actual Length of stay. The selector specificity, as well as predictive 
specificity, of .L.0. I. are. lower in post 75th percentiie subsét, 
The predictive specificity ‘decreases about 15%, but the selector 
specificity difference of 33% demonstrates the rapidly decreasing 
of L.0.C.1. after ch percentile for this criterion. it 
selects far more patients for review as the enniaiete: length of stay 
| increases. This factor is not surprising, as one would expect that the -_ 
service code patterns of appropriately located patients in this subset are 
not very different from patients needing further review and therefore 
are nine difficult to differentiate. Keeping in mind the intent of the 


selector model, errors of this type are still preferable. However, 


because the statistical significance of these differences Cannot be > 
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accuracy difference between the models. beOBs 


diagnoses, respectively. 
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‘determined, any subsequent assertions that are stated about L.0.C.I. 


will be for the pre 75th percentile subset of the evaluation set and 


will not necessarily hold true for L.0.C.I.'s abilities at any point in 


a patient's The results are equivalent whether the 


subset if defined by the current or the discharge diagnosis. 


em Comparison of the Correct Allocation Rates of the Selector Models 


The Correct Allocation Rate is a measure of the accuracy of a 


selector model. The Correct Allocation Rate for L.0.C.I. is 83% and for 


L.0.S. it is 54% indicating that L.0.C.I. is more accurate in its 


allocations. Because fewer days are eligible whén using L20;8.. this 


model selects a smaller total proportion of cases, a factor which can 


affect its accuracy. Therefore, the results of the sensitivity and 


specificity analysis will aid in interpreting 


and L.0.S., refer to 


C D 
the length of stay selector models using the current and discharge 


V.4.3 Selector Sensitivity of Models 


The proportion of True Positives, Pr(cJG,), is a measure of 


selector sensitivity. Although correction for Chance is not “included, a 


simple comparison of the selector sensitivity of two tests, or as in this 


case models, provides an indication of the increased sensitivity of one 
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test over the other: 


“2! 


Selector Sensitivity, 0 s./p = 21 


the use of this measure we conclude that L.0.C.1. is con- 
sensitive in selecting cases for review. For the 
observations the URNC review, L.0.C.Ie 
-would have selected 89% and 1.038. would select 21%. All observations 
in the set are eligible for sélection by L.0.C.1., whereas L.0.S. can 
‘only select 18%. Chance alone, could be nore sensitive. 

When comparing L.0.C.I. for only L.0.S. selected cases we Pind that the 
selector sensitivity of L.0.C.I. = .93. This result further amplifies 
"the selector sensitivity level of L.0.C.1. in that almost all (25 out 
; of 27) the True Positives in this group would have been selected by > 
L.0.C.I. of the 7D that L.0.S. would have missed, 88% would sides been 


selected by L.O.C.I. 


V.4.4' Selector Specificity of Models 


4 


The proportion of True Negatives, Pr(Gz | Gz) is a measure of | 


selector specificity. » The selector specificity reflects a model's 


discriminating powers. It indicates the model's error rate for observa- 


tions not necessitating further review. If the selector specificity. 
level is low then too many patients would be. reviewed unnecessarily | 


which eliminates the value of having a selection process. 
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Specificity, 87 


Selector Specificity, s./D = 


The results demonstrate that L.0.S. is more specific than 


-L.0.C.I. Again, this-is due in part to the proportion of cases which 


are eligible for selection by each method. However, when the selector © 


specificity of L.0.C.I. is calculated for L.0.S. selected cases, it is 


found to be 88%. For both selector models, the sélector specificity | 


levels are fairly high, a factor necessary for a selection process, 


V.4.5 Predictive Sensitivity and Specificity of Selector Models 


Selector. sensitivity and specificity assess the performance of 
the models, i.e., how good is the selection process, The predictive 
sensitivity and specificity are a reorganization of the data to assess 


the value of the information provided by the models. The informational 


_ value of both positive and negative responses by the selector model are 
important to evaluating the models' errors. AS already stated, these 


two measures are related to the selector sensitivity and specificity, 


however they are not as dependent on the proportion of cases eligible 


for selection. 


ond 


Predictive Sensitivity, 0 s./D = 
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| patient selected by L.O.S. Furthermore, patients not selected for 
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Predictive Specificity, 0 287 
Predictive Specificity, 0.8./C = 52 


Predictive Specificity, 4 = 252 


The predictive value of L.0.C.I. exceeds that of Le00Se for both 
positive and negative responses. A patient selected for review by 


L.O.C.I. has a higher probability of needing further review than a 


review by L.0.C.I. have a higher probability of not needing further 


review than those not selected by L.O.S. 


V.4.6 Summary of Comparison-of L.0.C.I. and L.0.S. 


The acceptable levels of senaitieles and specificity are’ 
dependent upon the intended iad of the test or model. : If the action 
associated with a positive result is’ custiy. tis time, risk, money, etc.) 
then high is If a False Negative 
result is costly neh is maximized. Keeping the objective: 
of Utilization Review and a selector of 
False Negatives with an acceptable level - False Positives would be the 


criterion for choosing a model. Assuming equal costs of errors, 


choosing the maximum accuracy rate is a means of considering both sensi- 


tivity and specificity; alternatively, the model with maximum predictive 


value of information can be the criterion for choice. If unequal costs 


é 


should be assumed, allocation errors or the specificity/sensitivity 


tradeoffs can be weighted in calculating a criterion’ score. 


When both selector sensitivity and selector specificity are taken 


j 
| 


into consideration, is preferable to for selecting cases 
for the URNC. Li0.0.2, ig far more sensitive and L.0.S. is only some= 
what more specific. When the predictive value, of ‘the information 

) provided by the models is considered, the preference sai L.O. C ole 
increases. Table 5. 14 summarizes the findings of the ‘evaluation of 
L.0.C.I. and L.O.S. Results are comparable for L.O.S. using either the 


* current or discharge diagnosis. 


A modification of the proposes models still yields comparable 
‘results. When patients who —" a discharge order written were deleted 
from the. evaluation set, ‘the ey level was 80% and 5 2% for L.0.C, i 
and O.Se, respectively. ‘who a discharge order Soule 
‘appear. on a separate list for veri fication of discharge ‘and reason for 
order not being carried out when that is the case. For a ste using 

L.0.C.I., patients who are appropriate in the acute 
for patients who are percentile, and who 
discharge order, should be verifies to ensure that the patient is not. 


‘ being discharged prematurely. | 


V.5 Outcome Analysis 


It has just been demonstrated that L.0.C.I. has greater 
predictive informational value than L.0.8., that L.0.C.I,. has far more 


selector sensitivity than L.0.S., but that L.0.S. is somewhat superior 


in selector specificity. These results were found when the criterion 


was the URNC judgment based on the computer medical record, 


For the entire study sample, the URNC sought information .in . 


addition to that provided by MIS 33% of the time before making a final 


UMI 


. 
ive) 


126 


e 
e 
{ 
| 


UMI 


| decision. In the subset of the evaluation set, she continued beyond 


comparable results for L.O.S. 


MIS | in the information acquisition sequence 25% of the time. When she 


added non-computer-based information, a total of 67% of the 


| observations were corisidered to be appropriately located in the acute | 


care hospital. Thus, for 83% of the subset, the URNC made a decision 


Ussng computer-based information. Are the measures of performance 


affected by the 17% change in the URNC's Becdstete? In view of the 


end be 0. Se the outcome analysis was 


C D’ 


The outcome has been eftned as the "confident" decision. ‘When 


the URNC consider's a patient as "APPROPRIATE" she takes no further 


action. When she considers the pation ‘as “NTNAPPROPRIATE", ‘she may 
take any of a number of different actions, a no action, but with : 


documented explanations Hence, all "confident" decisions are that the 


observation is either "APPROPRIATE" or "INAPPROPRIATE" in the acute 


hospital. The results of the outcome analysis. are summarized in 


Table 5015.6 
The selector; sesitivity levels. of L.0.C.I. and did 
premteay ve the "confident" decisions. The selector specificity of . 
to 60%, whereas .it only LeOeSe 
‘this indicates that L.0.C.I. selects more acest for review that the 
URNC considers "APPROPRIATE" when all available patient EP ere is . 
used. However, if we deduct the patients that L. 0. C. I. selected that 
the URNC arrived at a. "confident" decision veing computer-based 
information - she could quickly review the ‘gelectea patients by the use 


,of MIS - the selector specificity level increases to 81%. 
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_ When non-computer-based information is added, the predictive 
a positive response decreasés for both L.0.C.I. and 


however, it remains greater for ‘Le 0.C.I. The predictive value of a 


negative response for both models, but for it is 


much higher. If L.O. C. I. does not select a patient for review, there . 


isa 92% probability, that. the patient is truly appropriate. tn view of 


objective of maintaining delegated status, 


satisfactory. 


Summary 


The outcome analysis proccss results that are very, similar oa 
those found in. the. evaluation of the models for approximating a decision 
using computer-based information 0.0.35 allocations have a 


higher probability of.-being correct. For a review selector 


Bi peaches, missing patients who should be reviewed is moré serious than 
unnecessarily reviewing an appropriately located patient. of course, 


there isa tradeoff between the two types of errors and the tradeoff . 


ratio must be considered. L.O.S. misses 78% of the patients it should ° 


review against Le 0. C.I. missing 11%, which may be far more serious than 


the 40% which would be unnecessarily reviewed if using Le 0.C.I. versus 


the 15% erroneously selected by L.O.S. 


When evaluating the information provided by ‘the models, LsQsGsds 
is superior sai predictive value. wtstuttranlh it must be remembered that 


ne value is relative to the actions i are gr oi as a result of 


having or not having the information. | 
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CHAPTER VI CONCLUSIONS 


Introduction 


The PSRO program requires the review of hospitalized patients to 


assure the appropriate utilization of health care services while 


ensuring the quality of care. The selection of patients for review 
has been the subject of this study. To ascertain if the selection 


process could be improved, this dissertation proposed to investigate 


Hypothesis 1: Patients who may be inappropriately located 
in an acute hospital are. Characterized by a 
medical services profile (i.e. current orders) 


which differs from that of patients who are 


probably appropriately located. 


Hypothesis 2: Inappropriate utilization is more accurately 
predicted by a level of care index (L.0.C.I.) 


than by length of stay by diagnosis (LiOeBa}e. 


Hypothesis 3: When non-computer based information is. added, 
probable inappropriate location is more 
accurately predicted by a level of care 


selector model than by a length of stay 


selector model. 


‘ 
| 
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results of this experiment have demonstrated that the 
who are appropriately located in an acute. do have 
different medical services profiles from those patients ‘who are not 


appropriately located. 


A computerized decision model — the medical 
services profile as a set of predictor variables can ‘discriminate 


between those patients who may need to be in an acute hospital and those 


' who may not with an 81% level of accuracy. The computerized decision 
model is an index of the level of care needed by a patient, and can 
‘serve as a Selector model for Utilization Review. This ato 


index of level of care isa more valid predictor for a Utilization 


Review than the length of norms. 


of care index superior to the length of stay norms at selecting the 


“Not — the level. 


Patients the URNC would want to have selected for further Pare. but 
its performance | remains commaiebie at predicting. the confident 
decision --- the decision prompting feedback for corrective action, | 
Stated otherwise, the level of care index is better at predicting the 

-URNC'S confident the who are "APPROPRIATE" 
(probably locates) and "INAPPROPRIATE" (probably igappro- 


-_priately located). : 
Therefore, Bs can be stated that inappropriate utilization is 

more accurately predicted by the level of care index developed in this 

study, than by the length of ‘stay selector model proposed by the me 


Professional Standards Review Organizations. 


‘Beat th care experts defined a seas of 30 clusters of medical 


services capable of describing a patient day in terms Of current orders, 


Seven clusters of medical services were found to he useful predictors of 
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appropriate utilization. They are listed in Table 6.1 in descending 


order of discriminating powers based on the order the variables were 
selected sic a stepwise linear discriminant analysis. 

A discriminant function which is the sum a chite weighted | ; 
variables was empirically determined by a Logistic Discriminant Method 


and can predict which Sieben to select with an. 81% level of accuracy. 


‘The methodology. used not only provides a classification of the . 


hospitalized population, but also specifies the certainty with which 


each individual is Classified correctly.’ The combined effort of the . 


health care in the Sun jectivery defined 


medical services classification and clustering, and the power inherent 


in the statistical methodology used to enpiricsliy differentiate 
patterns, has yielded a mathematical . model ‘which approximates a comp lex | 


decision process.. 


VI.1 Implications 


* 


‘The endeavor undertaken in this research has been basically | 
methodological in nature. Had the resolution of the hypotheses posited 


been negative, some of the following issues would not be addressed. 


However, this not being the case, the costs and benefits of each model: . 


need to be evaluated. 


A cost effectiveness analysis of continued stay reviews was: per- 


formed by Averill and McMahon.* Their results indicate that the costs 


“associated with continued stay reviews are not likely to be compensated 


for by the potential They state that the 


Savings may be increased if the feedback and corrective action are more — 
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timely, that is decreasing the time for corrective action from 48 to 24 


hours. They also state that if the selection method chooses a higher 


proportion of patients who are inappropriately located, then the 


financial benefits would increase. However, they felt that if the 


selection process were more complex, it would be more costly, which | 


would in turn affect the potential financial benefits.” 


‘Their -pindines and assertions indicate the impor tance of: snes 


and accurate patient selection. Other studies support these assertions 


demonstrating that misutilized days can be decreased by efficacious feed- 


back procedures. 


The proposed level of care index ia timely because it reviews 


“every patient each day and it has a significantly higher probability of 


: identifying the patients who are inappropriately located. 


To evaluate the relative cost effectiveness of the proposed level 
of care index and the PSRO length of ~~ selector model, the following — 
factors need to be addressed. ; 

me costs should include: 

1). development and implementation of computer system of models, 

is progranming- costs for all the medical services, 
for converting a patient's medical services profile into a 
service cluster pattern, for the discriminant 


evaluation, for creating a data base of computer recognizable 


and sutamates assignment of lengths of 


operating and maintenance costs of each model, etc. 


| 
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2) cost of cinetwisiee review 
a) when the URNC can use only MIS fev deriving at a confident 
decision 
“b) when the URNC must use the chart to establish appropriate ? 
‘location. 
cost of missing a aay 
| a) when corrective action could have been initiated 
b) for documentation 


The benefits should include: 
prosoetion of coordifhator's time spent reviewing patients 
with a higher probability of action 
2) effect on likelihood of loss of delegated status 
3). effect on coordinator's decision imaking depending on 
criterion used for patient seldction 
4) effect on the eftustivenans of Utilization Review when 
a) providing feedback consistent with 
| physician decision making if the physician 
_ information on resources available tor patient care, or | 
| b) whens triggered by other criteria. 
5) accuracy of eeee on appropriate utilization of acute hospital 
and Seesena for inappropriate utilization due to availability 


of alternative facilities. . 


It was’not proposed that these questions would be answer ed here; 


it is strongly being suggested that this work be done. - 


VI.2 Conclusions 


An effective and efficient utilization review system is of con= 
cern to various indiyiduals and groups involved in the delivery of 
health care. Obviously, if it is not effective, it is not fulfilling 


its objective; if it is not efficient, it may not be justifying its 


raison d'étre,. 
Utilization Review is primarily a classification problem--that 


of classifying patients into the appropriate health care institution 


according to their medical care needs. Although peer review is dele- 


gated to ——e a physician representative can determine or screen 


: patients for physician review. The use of Utilization =e : 


Coordinators is one way of using petom time more efficiently, as 


well as increasing the Likelihood bach the review Process is more 


‘ effective.” Length of stay by diagnosis has been proposed ~~ the 
PSROs as the indicator to be used for deciding when to review a patient. 
A as been developed that can 
the Utilization Review Coordinator’ s judgments. It is an rsa of the 
level of care required by a bevel of care index is a 


‘better indicator for determining which patients should be reviewed for 


probable 
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NOTES 


Ve ‘U. Ss. Department of Health, Education, and Welfare, Office of 
Professional Standards Review, P.S.R.O. Program Manual, — 
Rockville, Maryland, 1974. 


2. Averill, R eF. and McMahon, L. F.' "A Cost Benefit Analysis of 
Continued Stay Certification," Medical 15:158, 


 Restuccia, J.D. The Effect of Feedback on Physicians' Decisions 
to Reduce Hospital Misutilization. Unpublished 
Dr. P.H. Dissertation, University of California 
School of Public Health, Berkeley, 1977. 
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APPENDIX A 


DICTIONARY OF A SAMPLE OF 


MEDICAL SERVICES AND ASSOCIATED CODES | | ; 


The medical services were printed with the niiaisiae aif the 
“Technicon MIS Corporation... 

I would like to thank the Technicon MIS peregree tee for their: 
conperative assistance. 
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APPENDIX 


UTILITY CURVES FOR SERVICE CODE CLUSTERS 


AS A FUNCTION OF AGE AND PRN/non-PRN FACTORS 
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AGE ~ Non-PRN (Number of days) ~ 


PIGURE 1,la UTILITY CURVE FOR AGE - Non-PRN OF CLUSTER, 
NURSING (Intensity 1, Intensity 3) 
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AGE - PRN (Number of days) 


FIGURE lyla UTILITY CURVE FOR AGE - PRN OF CLUSTER 
NURSING: (Intensity 1, Intensity 3) | 
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FIGURE 1,1q UTILITY CURVE FOR QUANTITY OF CLUSTER 


NURSING (Intensity 1, Intensity 3) 
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: FIGURE i. 2a UTILITY CURVE FOR AGE - Non=PRN OF en 
NURSING (Intensity 5) 


AGE ~ PRN (Number of days). 


FIGURE 1,2a UTILITY CURVE FOR AGE - PRN OF CLUSTER 
NURSING (Intensity 5) 
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FIGURE 1.2q UTILITY CURVE FOR QUANTITY OF CLUSTER 
NURSING (Intensity 5) 
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AGE - Non=PRN (Number of days) 


FIGURE 1,3a UTILITY CURVE FOR AGE ~ Non=PRN OF CLUSTER 
NURSING (Intensity 7) 
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FIGURE 1.3a UTILITY CURVE FOR AGE - PRN OF CLUSTER 
NURSING (Intensity 7). 
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PIGURE l. 4a UTILITY CURVE FOR AGE - Non=-PRN OF CLUSTER 
MEDICATIONS (Intensity 1, IV intensity. 
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FIGURE 1.4a UTILITY CURVE FOR AGE - PRN OF CLUSTER 
MEDICATIONS Cintensity ‘Iv Intensity 3) 
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PIGURE 1.4q UTILITY. CURVE FOR QUANTITY OF CLUSTER 
| MEDICATIONS (Intensity 1, IV Intensity 3) 
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AGE - Non=PRN (Number of days) 


FIGURE’ 1,5a UTILITY CURVE FOR AGE - Non-PRN OF: CLUSTER : 
MEDICATIONS. (Intensity 3 IM & PO, Intensity 5) Non-PRN 
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FIGURE 1.6a UTILITY CURVE FOR AGE - PRN OF CLUSTER 
MEDICATIONS (Intensity 3 IM & PO, Intensity 5) PRN 
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FIGURE 1.5q UTILITY CURVE FOR QUANTITY OF, CLUSTER , 
MEDICATIONS 3 IM & PO, Intensity 5) Non-PRN 
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..PIGURE 1. 6q UTILITY CURVE FOR QUANTITY OF CLUSTER 


MEDICATIONS. (Intensity 3 IM & PO, potaneiey 5) PRN 
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FIGURE 1.7a UTILITY CURVE FOR AGE ~ PRN OF CLUSTER 
MEDICATIONS (Intensity 7) 
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FIGURE 1.7q UTILITY CURVE FOR QUANTITY OF CLUSTER 
MEDICATIONS (Intensity 7) 
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FIGURE 1.8a UTILITY CURVE FOR AGE = Non-PRN OF CLUSTER 
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FIGURE 1,8a UTILITY CURVE FOR AGE - PRN OF CLUSTER 
ANCILLARY DIAGNOSTIC (Intensities 
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FIGURE 1.8q UTILITY CURVE FOR QUANTITY OF CLUSTER 
ANCILLARY DIAGNOSTIC (Intensities 1, 3, 5) 
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AGE - Non=PRN (Number of days) 
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FIGURE 1.9a UTILITY CURVE FOR AGE - Non-PRN OF CLUSTER | 
: ANCILLARY DIAGNOSTIC (Intensity 7) 
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FIGURE 1,9a “UTILITY CURVE OF AGE, = PRN OF CLUSTER 
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FIGURE 1.9q UTILITY CURVE FOR QUANTITY OF CLUSTER 
ANCILLARY DIAGNOSTIC {Intensity 7) 


187 
100 
| 80. 
U 60 
T 
I 
L 40 
I 
Y 20 | 
0 
2 4 eae — 12 14 16 18 20 


UMI 


AC. 


80. 


60 - 


40 - 


188 


AGE - Non=PRN (Number of days) 


FIGURE 1,10a UTILITY CURVE FOR AGE -Non=PRN OF CLUSTER | 
: ANCILLARY DIAGNOSTIC (Intensity 9) 
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FIGURE 1.10a UTILITY CURVE FOR AGE - PRN OF CLUSTER 


ANCILLARY DIAGNOSTIC (Intensity 9) 
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FIGURE 1,10q UTILITY CURVE FOR QUANTITY OF CLUSTER 
ANCILLARY DIAGNOSTIC (Intensity 9) | 
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FIGURE 1ylla UTILITY CURVE FOR AGE -, Non-PRN OF CLUSTER. 
ANCILLARY THERAPEUTIC (Intensity 1) 
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FIGURE 1s1la UTILITY CURVE FOR AGE - PRN OF CLUSTER 
- ANCILLARY THERAPEUTIC (Intensity 19 
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FIGURE 1,11q UTILITY CURVE FOR QUANTITY OF CLUSTER 
| ANCILLARY THERAPEUTIC (Intensity 1) . 
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| FIGURE 1, 128 UTILITY CURVE FOR AGE — Non=PRN OF CLUSTER 
ANCILLARY THERAPEUTIC (Intensity 3, ey 5) 
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FIGURE 1,124 UTILITY CURVE FOR AGE~ PRN OF. CLUSTER 
ANCILLARY THERAPEUTIC ‘(Intensity 3, Intensity 5) 
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FIGURE 1,12q UTILITY CURVE FOR QUANTITY OF CLUSTER 
. ANCILLARY THERAPEUTIC (Intensity 3, Intensity 5) | 
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AGE - Non-PRN (Number of days) 


FIGURE 1,13a UTILITY CURVE FOR AGE - Non=PRN OF CLUSTER 
~ ANCILLARY THERAPEUTIC (Intensity 7, Intensity 9) 
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FIGURE 1.13a UTILITY CURVE FOR AGE ~ PRN OF CLUSTER . 
ANCILLARY THERAPEUTIC (Intensity 7, Intensity 9) 
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PIGURE 1,13q UTILITY CURVE FOR QUANTITY OF CLUSTER 
| ANCILLARY THERAPEUTIC (Intensity 7, Intensity 9) 
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STEPWISE VARIABLE SELECTION 
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PLEASE NOTE: 


Appendix C is a computer tS. 

print-out. Type is light and . 

indistinct. Filmed in the 
way possible. 
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APPENDIX D 


SEVEN VARIABLE LOGISTIC 


- FUNCTION on ESTIMATION SET 
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APPENDIX E 


INDIVIOUAL RECORDS OF EVALUATION SET 


INCLUDING LEVEL OF: CARE INDEX 


(The Level of Care Index is the value of. the ‘logit’. 
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